Support for candidate components index

This commit adds a "spring-context-indexer" module that can be added to
any project in order to generate an index of candidate components defined
in the project.

`CandidateComponentsIndexer` is a standard annotation processor that
looks for source files with target annotations (typically `@Component`)
and references them in a `META-INF/spring.components` generated file.

Each entry in the index is the fully qualified name of a candidate
component and the comma-separated list of stereotypes that apply to that
candidate. A typical example of a stereotype is `@Component`. If a
project has a `com.example.FooService` annotated with `@Component` the
following `META-INF/spring.components` file is generated at compile time:

```
com.example.FooService=org.springframework.stereotype.Component
```

A new `@Indexed` annotation can be added on any annotation to instructs
the scanner to include a source file that contains that annotation. For
instance, `@Component` is meta-annotated with `@Indexed` now and adding
`@Indexed` to more annotation types will transparently improve the index
with additional information. This also works for interaces or parent
classes: adding `@Indexed` on a `Repository` base interface means that
the indexed can be queried for its implementation by using the fully
qualified name of the `Repository` interface.

The indexer also adds any class or interface that has a type-level
annotation from the `javax` package. This includes obviously JPA
(`@Entity` and related) but also CDI (`@Named`, `@ManagedBean`) and
servlet annotations (i.e. `@WebFilter`). These are meant to handle
cases where a component needs to identify candidates and use classpath
scanning currently.

If a `package-info.java` file exists, the package is registered using
a "package-info" stereotype.

Such files can later be reused by the `ApplicationContext` to avoid
using component scan. A global `CandidateComponentsIndex` can be easily
loaded from the current classpath using `CandidateComponentsIndexLoader`.

The core framework uses such infrastructure in two areas: to retrieve
the candidate `@Component`s and to build a default `PersistenceUnitInfo`.
Rather than scanning the classpath and using ASM to identify candidates,
the index is used if present.

As long as the include filters refer to an annotation that is directly
annotated with `@Indexed` or an assignable type that is directly
annotated with `@Indexed`, the index can be used since a dedicated entry
wil be present for that type. If any other unsupported include filter is
specified, we fallback on classpath scanning.

In case the index is incomplete or cannot be used, The
`spring.index.ignore` system property can be set to `true` or,
alternatively, in a "spring.properties" at the root of the classpath.

Issue: SPR-11890

spring-context/src/main/java/org/springframework/context/annotation/ClassPathScanningCandidateComponentProvider.java

@@ -18,6 +18,7 @@ package org.springframework.context.annotation;
 
 import java.io.IOException;
 import java.lang.annotation.Annotation;
+import java.util.HashSet;
 import java.util.LinkedHashSet;
 import java.util.LinkedList;
 import java.util.List;
@@ -28,9 +29,13 @@ import org.apache.commons.logging.LogFactory;
 
 import org.springframework.beans.factory.BeanDefinitionStoreException;
 import org.springframework.beans.factory.annotation.AnnotatedBeanDefinition;
+import org.springframework.beans.factory.annotation.AnnotatedGenericBeanDefinition;
 import org.springframework.beans.factory.config.BeanDefinition;
 import org.springframework.beans.factory.support.BeanDefinitionRegistry;
 import org.springframework.context.ResourceLoaderAware;
+import org.springframework.context.index.CandidateComponentsIndex;
+import org.springframework.context.index.CandidateComponentsIndexLoader;
+import org.springframework.core.annotation.AnnotationUtils;
 import org.springframework.core.env.Environment;
 import org.springframework.core.env.EnvironmentCapable;
 import org.springframework.core.env.StandardEnvironment;
@@ -43,17 +48,24 @@ import org.springframework.core.type.classreading.CachingMetadataReaderFactory;
 import org.springframework.core.type.classreading.MetadataReader;
 import org.springframework.core.type.classreading.MetadataReaderFactory;
 import org.springframework.core.type.filter.AnnotationTypeFilter;
+import org.springframework.core.type.filter.AssignableTypeFilter;
 import org.springframework.core.type.filter.TypeFilter;
 import org.springframework.stereotype.Component;
 import org.springframework.stereotype.Controller;
+import org.springframework.stereotype.Indexed;
 import org.springframework.stereotype.Repository;
 import org.springframework.stereotype.Service;
 import org.springframework.util.Assert;
 import org.springframework.util.ClassUtils;
 
 /**
- * A component provider that scans the classpath from a base package. It then
- * applies exclude and include filters to the resulting classes to find candidates.
+ * A component provider that provides candidate components from a base package. Can
+ * use {@link CandidateComponentsIndex the index} if it is available of scans the
+ * classpath otherwise. Candidate components are identified by applying exclude and
+ * include filters. {@link AnnotationTypeFilter}, {@link AssignableTypeFilter} include
+ * filters on an annotation/super-class that are annotated with {@link Indexed} are
+ * supported: if any other include filter is specified, the index is ignored and
+ * classpath scanning is used instead.
  *
  * <p>This implementation is based on Spring's
  * {@link org.springframework.core.type.classreading.MetadataReader MetadataReader}
@@ -63,10 +75,12 @@ import org.springframework.util.ClassUtils;
  * @author Juergen Hoeller
  * @author Ramnivas Laddad
  * @author Chris Beams
+ * @author Stephane Nicoll
  * @since 2.5
  * @see org.springframework.core.type.classreading.MetadataReaderFactory
  * @see org.springframework.core.type.AnnotationMetadata
  * @see ScannedGenericBeanDefinition
+ * @see CandidateComponentsIndex
  */
 public class ClassPathScanningCandidateComponentProvider implements EnvironmentCapable, ResourceLoaderAware {
 
@@ -81,6 +95,8 @@ public class ClassPathScanningCandidateComponentProvider implements EnvironmentC
 	private MetadataReaderFactory metadataReaderFactory =
 			new CachingMetadataReaderFactory(this.resourcePatternResolver);
 
+	private CandidateComponentsIndex componentsIndex;
+
 	private String resourcePattern = DEFAULT_RESOURCE_PATTERN;
 
 	private final List<TypeFilter> includeFilters = new LinkedList<>();
@@ -132,6 +148,7 @@ public class ClassPathScanningCandidateComponentProvider implements EnvironmentC
 	public void setResourceLoader(ResourceLoader resourceLoader) {
 		this.resourcePatternResolver = ResourcePatternUtils.getResourcePatternResolver(resourceLoader);
 		this.metadataReaderFactory = new CachingMetadataReaderFactory(resourceLoader);
+		this.componentsIndex = CandidateComponentsIndexLoader.loadIndex(resourceLoader.getClassLoader());
 	}
 
 	/**
@@ -263,6 +280,58 @@ public class ClassPathScanningCandidateComponentProvider implements EnvironmentC
 	 * @return a corresponding Set of autodetected bean definitions
 	 */
 	public Set<BeanDefinition> findCandidateComponents(String basePackage) {
+		if (isIndexSupported()) {
+			return addCandidateComponentsFromIndex(basePackage);
+		}
+		else {
+			return scanCandidateComponents(basePackage);
+		}
+	}
+
+	protected Set<BeanDefinition> addCandidateComponentsFromIndex(String basePackage) {
+		Set<BeanDefinition> candidates = new LinkedHashSet<>();
+		try {
+			Set<String> types = new HashSet<>();
+			for (TypeFilter filter : this.includeFilters) {
+				String stereotype = extractStereotype(filter);
+				if (stereotype == null) {
+					throw new IllegalArgumentException("Failed to extract stereotype from "+ filter);
+				}
+				types.addAll(this.componentsIndex.getCandidateTypes(basePackage, stereotype));
+			}
+			boolean traceEnabled = logger.isTraceEnabled();
+			boolean debugEnabled = logger.isDebugEnabled();
+			for (String type : types) {
+				MetadataReader metadataReader = this.metadataReaderFactory.getMetadataReader(type);
+				if (isCandidateComponent(metadataReader)) {
+					AnnotatedGenericBeanDefinition sbd = new AnnotatedGenericBeanDefinition(
+							metadataReader.getAnnotationMetadata());
+					if (isCandidateComponent(sbd)) {
+						if (debugEnabled) {
+							logger.debug("Using candidate component class from index: " + type);
+						}
+						candidates.add(sbd);
+					}
+					else {
+						if (debugEnabled) {
+							logger.debug("Ignored because not a concrete top-level class: " + type);
+						}
+					}
+				}
+				else {
+					if (traceEnabled) {
+						logger.trace("Ignored because matching an exclude filter: " + type);
+					}
+				}
+			}
+		}
+		catch (IOException ex) {
+			throw new BeanDefinitionStoreException("I/O failure during classpath scanning", ex);
+		}
+		return candidates;
+	}
+
+	protected Set<BeanDefinition> scanCandidateComponents(String basePackage) {
 		Set<BeanDefinition> candidates = new LinkedHashSet<>();
 		try {
 			String packageSearchPath = ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX +
@@ -374,6 +443,57 @@ public class ClassPathScanningCandidateComponentProvider implements EnvironmentC
 		return (beanDefinition.getMetadata().isConcrete() && beanDefinition.getMetadata().isIndependent());
 	}
 
+	/**
+	 * Determine if the index can be used by this instance.
+	 * @return {@code true} if the index is available and the configuration of this
+	 * instance is supported by it, {@code false otherwise}.
+	 */
+	protected boolean isIndexSupported() {
+		if (this.componentsIndex == null) {
+			return false;
+		}
+		for (TypeFilter includeFilter : this.includeFilters) {
+			if (!isIndexSupportsIncludeFilter(includeFilter)) {
+				return false;
+			}
+		}
+		return true;
+	}
+
+	/**
+	 * Determine if the specified include {@link TypeFilter} is supported by the index.
+	 * @param filter the filter to check
+	 * @return whether the index supports this include filter
+	 * @see #extractStereotype(TypeFilter)
+	 */
+	protected boolean isIndexSupportsIncludeFilter(TypeFilter filter) {
+		if (filter instanceof AnnotationTypeFilter) {
+			Class<? extends Annotation> annotation = ((AnnotationTypeFilter) filter).getAnnotationType();
+			return (AnnotationUtils.isAnnotationDeclaredLocally(Indexed.class, annotation)
+					|| annotation.getName().startsWith("javax."));
+		}
+		if (filter instanceof AssignableTypeFilter) {
+			Class<?> target = ((AssignableTypeFilter) filter).getTargetType();
+			return AnnotationUtils.isAnnotationDeclaredLocally(Indexed.class, target);
+		}
+		return false;
+	}
+
+	/**
+	 * Extract the stereotype to use for the specified compatible filter.
+	 * @param filter the filter to handle
+	 * @return the stereotype in the index matching this filter
+	 * @see #isIndexSupportsIncludeFilter(TypeFilter)
+	 */
+	protected String extractStereotype(TypeFilter filter) {
+		if (filter instanceof AnnotationTypeFilter) {
+			return ((AnnotationTypeFilter) filter).getAnnotationType().getName();
+		}
+		if (filter instanceof AssignableTypeFilter) {
+			return ((AssignableTypeFilter) filter).getTargetType().getName();
+		}
+		return null;
+	}
 
 	/**
 	 * Clear the underlying metadata cache, removing all cached class metadata.

spring-context/src/main/java/org/springframework/context/index/CandidateComponentsIndex.java

@@ -0,0 +1,87 @@
+/*
+ * Copyright 2002-2016 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.context.index;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.springframework.util.LinkedMultiValueMap;
+import org.springframework.util.MultiValueMap;
+
+/**
+ * Provide access to the candidates that are defined in {@code META-INF/spring.components}.
+ * <p>
+ * An arbitrary number of stereotypes can be registered (and queried) on the index: a
+ * typical example is the fully qualified name of an annotation that flags the class for
+ * a certain use case. The following call returns all the {@code @Component}
+ * <b>candidate</b> types for the {@code com.example} package (and its sub-packages):
+ * <pre class="code">
+ * Set&lt;String&gt; candidates = index.getCandidateTypes(
+ *         "com.example", "org.springframework.stereotype.Component");
+ * </pre>
+ *
+ * The {@code type} is usually the fully qualified name of a class, though this is
+ * not a rule. Similarly, the {@code stereotype} is usually the fully qualified name of
+ * a target type but it can be any marker really.
+ *
+ * @author Stephane Nicoll
+ * @since 5.0
+ */
+public class CandidateComponentsIndex {
+
+	private final MultiValueMap<String, String> index;
+
+	CandidateComponentsIndex(List<Properties> content) {
+		this.index = parseIndex(content);
+	}
+
+	/**
+	 * Return the candidate types that are associated with the specified stereotype.
+	 * @param basePackage the package to check for candidates
+	 * @param stereotype the stereotype to use
+	 * @return the candidate types associated with the specified {@code stereotype}
+	 * or an empty set if none has been found for the specified {@code basePackage}
+	 */
+	public Set<String> getCandidateTypes(String basePackage, String stereotype) {
+		List<String> candidates = this.index.get(stereotype);
+		if (candidates != null) {
+			return candidates.parallelStream()
+					.filter(t -> t.startsWith(basePackage))
+					.collect(Collectors.toSet());
+		}
+		return Collections.emptySet();
+	}
+
+	private static MultiValueMap<String, String> parseIndex(List<Properties> content) {
+		MultiValueMap<String, String> index = new LinkedMultiValueMap<>();
+		for (Properties entry : content) {
+			for (Map.Entry<Object, Object> entries : entry.entrySet()) {
+				String type = (String) entries.getKey();
+				String[] stereotypes = ((String) entries.getValue()).split(",");
+				for (String stereotype : stereotypes) {
+					index.add(stereotype, type);
+				}
+			}
+		}
+		return index;
+	}
+
+}

spring-context/src/main/java/org/springframework/context/index/CandidateComponentsIndexLoader.java

@@ -0,0 +1,115 @@
+/*
+ * Copyright 2002-2016 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.context.index;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import org.springframework.core.SpringProperties;
+import org.springframework.core.io.UrlResource;
+import org.springframework.core.io.support.PropertiesLoaderUtils;
+import org.springframework.util.ConcurrentReferenceHashMap;
+
+/**
+ * Candidate components index loading mechanism for internal use within the framework.
+ *
+ * @author Stephane Nicoll
+ * @since 5.0
+ */
+public class CandidateComponentsIndexLoader {
+
+
+	private static final Log logger = LogFactory.getLog(CandidateComponentsIndexLoader.class);
+
+	/**
+	 * System property that instructs Spring to ignore the index, i.e.
+	 * to always return {@code null} from {@link #loadIndex(ClassLoader)}.
+	 * <p>The default is "false", allowing for regular use of the index. Switching this
+	 * flag to {@code true} fulfills a corner case scenario when an index is partially
+	 * available for some libraries (or use cases) but couldn't be built for the whole
+	 * application. In this case, the application context fallbacks to a regular
+	 * classpath arrangement (i.e. as no index was present at all).
+	 */
+	public static final String IGNORE_INDEX = "spring.index.ignore";
+
+	private static final boolean shouldIgnoreIndex =
+			SpringProperties.getFlag(IGNORE_INDEX);
+
+
+	/**
+	 * The location to look for components.
+	 * <p>Can be present in multiple JAR files.
+	 */
+	public static final String COMPONENTS_RESOURCE_LOCATION = "META-INF/spring.components";
+
+	private static final ConcurrentMap<ClassLoader, CandidateComponentsIndex> cache
+			= new ConcurrentReferenceHashMap<>();
+
+
+	/**
+	 * Load and instantiate the {@link CandidateComponentsIndex} from
+	 * {@value #COMPONENTS_RESOURCE_LOCATION}, using the given class loader. If no
+	 * index is available, return {@code null}.
+	 * @param classLoader the ClassLoader to use for loading (can be {@code null} to use the default)
+	 * @return the index to use or {@code null} if no index was found
+	 * @throws IllegalArgumentException if any module index cannot
+	 * be loaded or if an error occurs while creating {@link CandidateComponentsIndex}
+	 */
+	public static CandidateComponentsIndex loadIndex(ClassLoader classLoader) {
+		ClassLoader classLoaderToUse = classLoader;
+		if (classLoaderToUse == null) {
+			classLoaderToUse = CandidateComponentsIndexLoader.class.getClassLoader();
+		}
+		return cache.computeIfAbsent(classLoaderToUse, CandidateComponentsIndexLoader::doLoadIndex);
+	}
+
+	private static CandidateComponentsIndex doLoadIndex(ClassLoader classLoader) {
+		if (shouldIgnoreIndex) {
+			return null;
+		}
+		try {
+			Enumeration<URL> urls = classLoader.getResources(COMPONENTS_RESOURCE_LOCATION);
+			if (!urls.hasMoreElements()) {
+				return null;
+			}
+			List<Properties> result = new ArrayList<>();
+			while (urls.hasMoreElements()) {
+				URL url = urls.nextElement();
+				Properties properties = PropertiesLoaderUtils.loadProperties(new UrlResource(url));
+				result.add(properties);
+			}
+			if (logger.isTraceEnabled()) {
+				logger.trace("Loaded " + result.size() + "] index(es)");
+			}
+			int totalCount = result.stream().mapToInt(Properties::size).sum();
+			return (totalCount > 0 ? new CandidateComponentsIndex(result) : null);
+		}
+		catch (IOException ex) {
+			throw new IllegalArgumentException("Unable to load indexes from location ["
+					+ COMPONENTS_RESOURCE_LOCATION + "]", ex);
+		}
+	}
+
+}

spring-context/src/main/java/org/springframework/context/index/package-info.java

@@ -0,0 +1,20 @@
+/*
+ * Copyright 2002-2016 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Support package for reading and managing the components index.
+ */
+package org.springframework.context.index;

spring-context/src/main/java/org/springframework/stereotype/Component.java

@@ -42,6 +42,7 @@ import java.lang.annotation.Target;
 @Target(ElementType.TYPE)
 @Retention(RetentionPolicy.RUNTIME)
 @Documented
+@Indexed
 public @interface Component {
 
 	/**

spring-context/src/main/java/org/springframework/stereotype/Indexed.java

@@ -0,0 +1,87 @@
+/*
+ * Copyright 2002-2016 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.stereotype;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Indicate that the annotated element represents a stereotype for the index.
+ * <p>
+ * The {@code CandidateComponentsIndex} is an alternative to classpath
+ * scanning that uses a metadata file generated at compilation time. The
+ * index allows retrieving the candidate components (i.e. fully qualified
+ * name) based on a stereotype. This annotation instructs the generator to
+ * index the element on which the annotated element is present or if it
+ * implements or extends from the annotated element. The stereotype is the
+ * fully qualified name of the annotated element.
+ * <p>
+ * Consider the default {@link Component} annotation that is meta-annotated
+ * with this annotation. If a component is annotated with {@link Component},
+ * an entry for that component will be added to the index using the
+ * {@code org.springframework.stereotype.Component} stereotype.
+ * <p>
+ * This annotation is also honored on meta-annotations. Consider this
+ * custom annotation:
+ * <pre class="code">
+ * package com.example;
+ *
+ * &#064;Target(ElementType.TYPE)
+ * &#064;Retention(RetentionPolicy.RUNTIME)
+ * &#064;Documented
+ * &#064;Indexed
+ * &#064;Service
+ * public @interface PrivilegedService { ... }
+ * </pre>
+ * If this annotation is present on an type, it will be indexed with two
+ * stereotypes: {@code org.springframework.stereotype.Component} and
+ * {@code com.example.PrivilegedService}. While {@link Service} isn't directly
+ * annotated with {@code Indexed}, it is meta-annotated with {@link Component}.
+ * <p>
+ * It is also possible to index all implementations of a certain interface or
+ * all the sub-classes of a given class by adding {@code @Indexed} on it.
+ * Consider this base interface:
+ * <pre class="code">
+ * package com.example;
+ *
+ * &#064;Indexed
+ * public interface AdminService { ... }
+ * </pre>
+ * Now, consider an implementation of this {@code AdminService} somewhere:
+ * <pre class="code">
+ * package com.example.foo;
+ *
+ * import com.example.AdminService;
+ *
+ * public class ConfigurationAdminService implements AdminService { ... }
+ * </pre>
+ * Because this class implements an interface that is indexed, it will be
+ * automatically included with the {@code com.example.AdminService} stereotype.
+ * If there are more {@code @Indexed} interfaces and/or super classes in the
+ * hierarchy, the class will map to all their stereotypes.
+ *
+ * @author Stephane Nicoll
+ * @since 4.3.3
+ */
+@Target(ElementType.TYPE)
+@Retention(RetentionPolicy.RUNTIME)
+@Documented
+public @interface Indexed {
+}