The title explains the preconditions:
- spring beans (conforming to an interface), using @Transactional (or <tx:advice>)
- custom exception is thrown from within the service methods
There is a little trick – if the exception is unckecked (extends RuntimeException), it is being wrapped in a TransactionRollbackException.
So if you are expecting a custom exception (in a unit test, for example), it must be checked. Which is completely logical, of course, but people are often tempted by the easy usage of unchecked exceptions (no explicit need to declare or catch them).