diff --git a/.cursor/rules/building.mdc b/.cursor/rules/building.mdc new file mode 100644 index 000000000000..7ec9fabd47c4 --- /dev/null +++ b/.cursor/rules/building.mdc @@ -0,0 +1,32 @@ +--- +description: Maven build instructions for the NiFi codebase +alwaysApply: true +--- + + +# Building + +NiFi is a complex Maven codebase. Never build code (testing or otherwise) using javac. +Always use `mvn` instead, or preferably the `.mvnw` wrapper script. + +Additionally, building a maven module using the also-make flag (`-am`) is often very +expensive and slow. Instead, only build the specific module you are modifying. Assume that +the user has already built the entire codebase and that only the specific module you are +modifying needs to be built again. If this fails, you can prompt the user to build the entire +codebase, but only after you have attempted to build the relevant modules yourself first. +It is important not to run `mvn clean` at the root level or at the `nifi-assembly` level without +the user's express permission, as this may delete a running instance of NiFi, causing permanent +loss of flows and configuration. diff --git a/.cursor/rules/code-style.mdc b/.cursor/rules/code-style.mdc new file mode 100644 index 000000000000..5432070ecf71 --- /dev/null +++ b/.cursor/rules/code-style.mdc @@ -0,0 +1,88 @@ +--- +description: Java code style conventions for the NiFi codebase +globs: "**/*.java" +alwaysApply: false +--- + + +# Code Style + +NiFi adheres to a few code styles that are not necessarily common. Please ensure that you +observe these code styles. + +1. Any variable that can be marked `final` must be marked `final`. This includes + declarations of Exceptions, method arguments, local variables, member variables, etc. +2. Short-hand is highly discouraged in names of variables, classes, methods, etc., as well + as in documentation. Exceptions to this include in the framework, you may see references to + `procNode` for `ProcessorNode` or other such short-hand that is very difficult to confuse with + other terms, and it is used only when clearly defined such as `final ProcessorNode procNode = ...`. + Even though, however, we would not abbreviate `ControllerService` as `cs` because `cs` is too vague + and easily misunderstood. Instead, a value of `serviceNode` might be used. +3. Private / helper methods should not be placed before the first public/protected method + that calls it. +4. Unless the method is to be heavily reused, avoid creating trivial 1-2 line methods and + instead just place the code inline. +5. Code is allowed to be up to 200 characters wide. Avoid breaking lines into many short lines. +6. Avoid creating private methods that are called only once unless they are at least 10 + lines long or are complex. +7. It is never acceptable to use star imports. Import each individual class that is to be used. +8. Never use underscores in class names, variables, or filenames. +9. Never use System.out.println but instead use SLF4J Loggers. +10. Avoid excessive whitespace in method invocations. For example, instead of writing: + +```java +myObject.doSomething( + arg1, + arg2, + arg3, + arg4, + arg5 +); +``` + +Write this instead: + +```java +myObject.doSomething(arg1, arg2, arg3, arg4, arg5); +``` + +It is okay to use many newlines in a builder pattern, such as: +```java +final MyObject myObject = MyObject.builder() + .arg1(arg1) + .arg2(arg2) + .arg3(arg3) + .build(); +``` + +It is also acceptable when chaining methods in a functional style such as: +```java +final List result = myList.stream() + .filter(s -> s.startsWith("A")) + .map(String::toUpperCase) + .toList(); +``` + +11. When possible, prefer importing a class, rather than using fully qualified classname + inline in the code. +12. Avoid statically importing methods, except in methods that are frequently used in testing + frameworks, such as the `Assertions` and `Mockito` classes. +13. Avoid trailing whitespace at the end of lines, especially in blank lines. +14. The `var` keyword is never allowed in the codebase. Always explicitly declare the type of variables. +15. Prefer procedural code over functional code. For example, prefer using a for loop instead of a stream + when the logic is not simple and straightforward. The stream API is powerful but can be difficult to + read when overused or used in complex scenarios. Functional style is best used when the logic is simple + and chains together no more than 3-4 operations. diff --git a/.cursor/rules/ending-conditions.mdc b/.cursor/rules/ending-conditions.mdc new file mode 100644 index 000000000000..8242bb340057 --- /dev/null +++ b/.cursor/rules/ending-conditions.mdc @@ -0,0 +1,44 @@ +--- +description: Task completion checklist that must be verified before considering any task done +alwaysApply: true +--- + + +# Ending Conditions + +When you have completed a task, ensure that you have verified the following: + +1. All code compiles and builds successfully using `mvn`. +2. All relevant unit tests pass successfully using `mvn`. +3. All code adheres to the Code Style rules. +4. Checkstyle and PMD pass successfully using + `mvn checkstyle:check pmd:check -T 1C` from the appropriate directory. +5. Unit tests have been added to verify the functionality of any sufficiently complex method. +6. A system test or an integration test has been added if the change makes significant + changes to the framework and the interaction between a significant number of classes. +7. You have performed a full review of the code to ensure that there are no logical errors + and that the code is not duplicative or difficult to understand. If you find any code that + is in need of refactoring due to clarity or duplication, you should report this to the user + and offer to make those changes as well. +8. If creating a new Processor or Controller Service, ensure that all relevant annotations + have been added, including `@Tags`, `@CapabilityDescription`, `@UseCase`, and + `@MultiProcessorUseCase` as appropriate. + + +Do not consider the task complete until all of the above conditions have been met. When you +do consider the task complete, provide a summary of what you changed and which tests were +added or modified and what the behavior is that they verify. Additionally, provide any feedback +about your work that may need further review or that is not entirely complete. diff --git a/.cursor/rules/extension-development.mdc b/.cursor/rules/extension-development.mdc new file mode 100644 index 000000000000..614bc79b09eb --- /dev/null +++ b/.cursor/rules/extension-development.mdc @@ -0,0 +1,121 @@ +--- +description: Development patterns for NiFi extensions (Processors, Controller Services, Connectors). Covers Property Descriptors, Relationships, and common patterns. +alwaysApply: false +--- + + +# Extension Development + +This rule applies when developing NiFi extensions: Processors, Controller Services, and Connectors. + +## Property Descriptors + +Property Descriptors are defined as `static final` fields on the component class using +`PropertyDescriptor.Builder`. + +- **Naming:** Use clear, descriptive names. The `displayName` field should never be used. Make the + name itself clear and concise. Use Title Case for property names. +- **Required vs. optional:** Mark properties as `.required(true)` when the component cannot + function without them. Prefer sensible defaults via `.defaultValue(...)` when possible. + When a default value is provided, the property will always have a value. The `required` flag in this + case is more of a documentation aid to indicate the importance of the property. +- **Validators:** Always attach an appropriate `Validator` (e.g., `StandardValidators.NON_EMPTY_VALIDATOR`, + `StandardValidators.POSITIVE_INTEGER_VALIDATOR`). The Validator can be left off only when Allowable Values + are provided. In this case, do not include a Validator because it is redundant and confusing. +- **Expression Language:** If a property should support Expression Language, add + `.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)` or the + appropriate scope. Always document when Expression Language is supported in the property + description. Some developers tend to go overboard here and feel like Expression Language should be supported + everywhere, but this is a mistake! The default assumption should be that Expression Language is not supported + unless the value is expected to be different for every FlowFile that is processed. +- **Dependencies:** Use `.dependsOn(...)` to conditionally show properties based on the + values of other properties. This keeps the configuration UI clean and avoids exposing + irrelevant properties. If there is a dependency, it is important to understand that `.required(true)` means that + this property is required IF AND ONLY IF the dependency condition is met. + +## Processors + +- The `onTrigger` method should be focused on processing FlowFiles. Keep setup and teardown + logic in lifecycle methods when possible. +- Prefer `session.read()` and `session.write()` with callbacks over directly working with + streams to ensure proper resource management. +- Prefer `session.commitAsync()` over `session.commit()`. The `commit` method was the original implementation, + but it has now been deprecated in favor of `commitAsync`. The `commitAsync` call provide a clearer, cleaner + interface for handling post-commit actions including success and failure callbacks. In addition, the async + method allows Processors to be used much more efficiently in a Stateless NiFi flow. + +### Processor Lifecycle Annotations + +- Use `@OnScheduled` for setup that should happen once before the processor starts + running (e.g., creating clients, compiling patterns). +- Use `@OnStopped` for cleanup (e.g., closing clients, releasing resources). +- `@OnUnscheduled` is rarely used but can be used to interrupt long-running processes when the Processor is stopped. + Generally, though, it is preferable to write the Processor in such a way that long-running processes check `isScheduled()` + and stop gracefully if the return value is `false`. + +### Relationships +- **Declaration**: Relationships are defined as `static final` fields using `new Relationship.Builder()`. + Relationship names should generally be lowercase. +- **Success and Failure:** Most processors define at least a `success` and `failure` + relationship. Use `REL_SUCCESS` and `REL_FAILURE` as constant names. +- **Original relationship:** Processors that enrich or fork FlowFiles often include an + `original` relationship for the unmodified input FlowFile. + +### Use Case Documentation +The `@UseCase` and `@MultiProcessorUseCase` annotations help document common usage patterns for Processors. +This is helpful for users to understand when and how to use the component effectively. It is equally important +for Agents that can determine which components should be used for a given task. + +- Use `@UseCase` to document common use cases for the Processor. This helps users understand + when and how to use the component effectively. This is unnecessary for Processors that serve a single use case + that is clearly described by the component name and description. For example, a Processor that consumes messages + from a specific service likely does not need a `@UseCase` annotation because its purpose is clear. +- Use `@MultiProcessorUseCase` to document well-known patterns that involve multiple Processors working + together to achieve a common goal. Examples include List/Fetch patterns, Fork/Join patterns, etc. + The `@MultiProcessorUseCase` annotation should not be added to each individual Processor involved in the pattern. + Rather, the convention is to add the annotation to the last Processor in the flow that completes the pattern. + Some Processors will have one or more `@UseCase` annotations and no `@MultiProcessorUseCase` annotations, + while some will have one or more `@MultiProcessorUseCase` annotations and no `@UseCase` annotations. + + +## Controller Services + +Controller Services are objects that can be shared across multiple components. This is typically done for +clients that connect to external systems in order to avoid creating many connections, or in order to share +configuration across multiple components without the user having to duplicate configuration. Controller Services +can also be helpful for abstracting away some piece of functionality into a separate extension point so that the +implementation can be swapped out by the user. For example, Record Readers and Writers are implemented as Controller +Services so that the user can simply choose which format they want to read and write in a flexible and reusable way. + +That said, Controller Services can be more onerous to configure and maintain for the user, so they should +be used sparingly and only when there is a clear benefit to doing so. + +### Controller Service Lifecycle Annotations + +- Use `@OnScheduled` for setup that should happen once before the service is enabled (e.g., creating clients, compiling patterns). +- Use `@OnDisabled` for cleanup (e.g., closing clients, releasing resources). + + +## General Patterns + +- Use `ComponentLog` (obtained via `getLogger()`) for all logging, not SLF4J directly. + This ensures log messages are associated with the component instance and that they generate Bulletins. +- Use `@CapabilityDescription` to provide a clear and concise description of what the component does. This should not + be used for configuration details. +- Use `@Tags` to provide relevant keywords that help users find the component. +- Use `@SeeAlso` to reference related components. +- Use `@WritesAttributes` and `@ReadsAttributes` to document which FlowFile attributes are read and written by the component. +- Use `@DynamicProperty` to document any dynamic properties supported by the component. diff --git a/.cursor/rules/extension-testing.mdc b/.cursor/rules/extension-testing.mdc new file mode 100644 index 000000000000..e5143336f38c --- /dev/null +++ b/.cursor/rules/extension-testing.mdc @@ -0,0 +1,94 @@ +--- +description: Testing guidance for NiFi extensions (Processors, Controller Services, Connectors). Covers nifi-mock and TestRunner usage. +alwaysApply: false +--- + + +# Extension Testing + +This rule applies when writing tests for NiFi extensions: Processors, Controller Services, and Connectors. + +## Unit Tests + +Unit tests should be used to test individual classes and methods in isolation. This often +will result in mocking dependency classes. However, if there already exists a Mock +implementation of an interface or dependency class, it is preferred to use the existing +Mock implementation. Similarly, for simple classes, it is preferable to make use of the +real implementation of a class rather than creating a Mock implementation. We are infinitely +more interested in having tests that are fast, reliable, correct, and easy to maintain than +we are in having tests that adhere to strict and arbitrary definitions of what constitutes +a "unit test." + +## Use nifi-mock + +Tests for extensions should always make use of the `nifi-mock` mocking framework. This is +done through the `TestRunner` interface and its standard implementation, obtained via +`TestRunners.newTestRunner(processor)`. + +The `TestRunner` provides methods for: +- Setting property values (`setProperty`) +- Enqueueing FlowFiles (`enqueue`) +- Running the processor (`run`) +- Asserting transfer to relationships (`assertTransferCount`, `assertAllFlowFilesTransferred`) +- Validating processor configuration (`assertValid`, `assertNotValid`) +- Asserting content and attributes of FlowFiles (`assertContentEquals`, `assertAttributeEquals`, etc.) + +## No System Tests for Extensions + +System tests are not expected for extensions. Extensions are tested at the unit level using +`nifi-mock`. The `nifi-mock` framework provides sufficient isolation and simulation of the +NiFi runtime environment. + +## What to Test + +- **Property validation:** If the extension has a custom Validator, it +- **customValidate:** If the extension overrides the `customValidate` method, test that it correctly + validates the configuration and produces appropriate validation results. +- **Relationship routing:** Verify that FlowFiles are routed to the correct relationship + based on input and configuration. +- **Content transformation:** For processors that modify FlowFile content, verify that + output content matches expectations. +- **Attribute handling:** Verify that expected attributes are set on output FlowFiles. +- **Error handling:** Verify that error conditions (bad input, misconfiguration, simulated + failures) are handled correctly, typically by routing to a failure relationship. + +## What NOT to Test + +- **NiFi framework behavior:** Do not attempt to test the behavior of the NiFi framework itself. + For example, do not test that `session.commitAsync()` actually commits a transaction. Instead, + focus on testing that your extension behaves correctly when `commitAsync` is called, and trust + that the NiFi framework will handle the commit correctly. +- **Validator behavior:** If a custom validator is used by an extension, that custom validator should + be tested separate as a unit test for the validator itself. However, if the extension point provides + a `customValidate` method, that should absolutely be tested as part of the extension's unit tests. +- **The PropertyDescriptors that are returned:** Do not test that the `getSupportedPropertyDescriptors` + method returns the expected PropertyDescriptors. This is an anti-pattern because it does not properly + test that the extension abides by the contract of the API. For example, if a new PropertyDescriptor is + added whose default is to behave the same way as the old behavior, the test should absolutely pass. + However, if the test is written to expect a specific set of PropertyDescriptors, then the test will fail, + leading to confusion and unnecessary maintenance. + +## Controller Service Testing + +When a processor depends on a Controller Service, use `TestRunner.addControllerService` +and `TestRunner.enableControllerService` to wire up either a real or mock implementation +of the service for testing. + +## TestContainers + +For Processors that interact with external systems, it can be helpful to use TestContainers to spin up +a temporary instance of the external system for testing. This allows for more realistic integration tests +without requiring the user to have the external system installed and running on their machine. diff --git a/.cursor/rules/framework-testing.mdc b/.cursor/rules/framework-testing.mdc new file mode 100644 index 000000000000..3343b0d0102e --- /dev/null +++ b/.cursor/rules/framework-testing.mdc @@ -0,0 +1,60 @@ +--- +description: Testing guidance for NiFi framework code (not extensions). Covers when to use unit, integration, and system tests for framework classes. +alwaysApply: false +--- + + +# Framework Testing + +This rule applies when working on NiFi framework code (not Processors, Controller +Services, or Connectors). + +## Unit Tests + +Unit tests should be used to test individual classes and methods in isolation. This often +will result in mocking dependency classes. However, if there already exists a Mock +implementation of an interface or dependency class, it is preferred to use the existing +Mock implementation. Similarly, for simple classes, it is preferable to make use of the +real implementation of a class rather than creating a Mock implementation. We are infinitely +more interested in having tests that are fast, reliable, correct, and easy to maintain than +we are in having tests that adhere to strict and arbitrary definitions of what constitutes +a "unit test." + +## Integration Tests + +When working in the framework, unit tests are still important, but integration tests and +system tests are often more important. Integration tests are still allowed to use mocks but +typically we prefer to use real implementations of classes in order to ensure a more +realistic and holistic test. + +## System Tests + +System tests live in the `nifi-system-tests` module and should be used for any changes +that make significant changes to the framework and the interaction between a significant +number of classes. They should also be used for any changes that may be fairly isolated but +which are in a critical path of the framework, especially those that affect how data is +persisted, processed, or accessed; or those that affect how components are created, +configured, scheduled, or executed. + +Good candidates for system tests include changes to `ProcessScheduler`, `ProcessorNode`, +`ControllerServiceNode`, `FlowController`, `FlowManager`, how Parameters are handled, flow +synchronization, the repositories, etc. + +## Escalation + +Any unit test that ends up requiring a large number of mocks is a good candidate for an +integration test, and any integration test that ends up requiring a large number of mocks +is a good candidate for a system test. diff --git a/.cursor/rules/persona.mdc b/.cursor/rules/persona.mdc new file mode 100644 index 000000000000..d741efe5d992 --- /dev/null +++ b/.cursor/rules/persona.mdc @@ -0,0 +1,30 @@ +--- +description: AI persona and general approach for working on the Apache NiFi codebase +alwaysApply: true +--- + + +# AI Persona + +Act as an experienced Java software engineer. When considering how to implement a task, +first consider the big picture of what is being asked. Then determine which classes will +need to be updated. + +Quite often, a single request will require manipulating many different classes. Generally +speaking, it is best to avoid changing established interfaces, especially those in nifi-api. +It is acceptable when necessary, but any change in nifi-api needs to be backward compatible. +For example, you might introduce a new method with a default implementation, or add a new method +and deprecate an old one without removing it. diff --git a/.cursor/rules/testing-standards.mdc b/.cursor/rules/testing-standards.mdc new file mode 100644 index 000000000000..676b099dd70e --- /dev/null +++ b/.cursor/rules/testing-standards.mdc @@ -0,0 +1,95 @@ +--- +description: Shared test coding conventions for all NiFi automated tests (framework and extensions) +globs: "**/src/test/**/*.java" +alwaysApply: false +--- + + +# Testing Standards + +In addition to the general rules defined in the Code Style rule, follow these rules when +creating or manipulating automated tests. + +## Test Code Conventions + +1. NEVER add comments such as `// Given`, `// When`, `// Then`. These comments are + considered an anti-pattern and should be removed or replaced whenever they are + encountered. Instead, leave them out all together (preferred) or use comments that + clearly articulate what is happening, such as `// Setup`, `// Invoke method`, + `// Assert expected results`. One of the reasons that this is considered an anti-pattern + (in addition to the fact that the given/when/then nomenclature itself provides no + meaning) is that it assumes a very specific pattern in unit tests, that we will create a + bunch of objects, invoke the method we care about, make assertions, and then end. This + often results in many tests that are extremely repetitive. Instead, whenever it makes + sense to do so, create the prerequisite objects, invoke the method we care about with + appropriate arguments, make assertions, and then invoke again with a different set of + arguments, make assertions, etc. There is no need to have many repetitive methods that + each create many repetitive objects. + +2. Unit tests are Java. They are not English. As such, they should be written like Java. + Frameworks such as assertj that strive to make the unit tests look more "English-like" + should be avoided. Use of these frameworks sometimes works well but often quickly + devolves into automated tests that read like neither English nor Java. + +3. Like any other code, unit tests should be created using reusable methods where + appropriate. Do not create 15 methods that are all very similar and repetitive. Instead, + create reusable methods that can be called from each of the methods. + +4. Never use the `assert` keyword. Use JUnit assertions instead. + +5. Never create a test file that is centered around testing a method or capability. Unit + tests must always be named after the class they are testing. It is okay if a given unit + test class is very, very long. + +6. This is a Java project using the Maven structure. Java test files must always fall under + src/test/java of the appropriate sub-module. + +7. Never use pointless assertions such as assertDoesNotThrow. This adds nothing but + complexity. Just call the method, and if it throws an Exception, the test will fail. It + is assumed by default that each line does not throw an Exception. + +8. Avoid providing messages in assert statements when they do not offer benefits over the default message. + For example, rather than: +``` + assertEquals(3, processors.size(), "Expected 3 processors in the initial flow"); +``` + Just use: +``` + assertEquals(3, processors.size()); +``` +Because the expectation is already clear from assertion, and this message is actually harmful because it hides the size +of the `processors` Collection. + +## General Testing Philosophy + +- Unit tests should be used to verify any sufficiently complex method in a class. We should + *NOT* have unit tests for trivial methods such as getters and setters, or methods that are + only a few lines long and are not complex. A good general rule of thumb is that if a + person can understand a method and verify that it is correct in a few seconds, then it is + not necessary to have a unit test for that method. + +- Avoid adding multiple tests that are redundant. For example, if you have a method that + takes a single `String` argument it may make sense to test with `null`, a 0-length + string, and a long String. But do not test with 4 different Strings each with a different + value but which test the same lines of code. Avoid over-testing by adding multiple tests + whose differences are conditions that do not truly affect the behavior of the code. + +- Unit tests should always focus on the "contract" of the method. That is, given a certain + input, the test should assert that it receives the correct output. The test should NOT be + focused on the inner details of how the method works. Focusing on the inner details of + how the method works is an anti-pattern because it results in tests that are more brittle, + more difficult to maintain, and tests that fail when the method is improved or refactored + but still adheres to the same contract. diff --git a/.gitignore b/.gitignore index ff8a118e9a59..e299e15e43c4 100644 --- a/.gitignore +++ b/.gitignore @@ -21,7 +21,7 @@ nb-configuration.xml .java-version /nifi-nar-bundles/nifi-py4j-bundle/nifi-python-extension-api/src/main/python/dist/ __pycache__ -.cursor/ +.cursor/debug.log # Develocity .mvn/.develocity/ diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java index 1bb5fb07d982..38bd55573e59 100644 --- a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ComponentManifest.java @@ -33,6 +33,7 @@ public class ComponentManifest implements Serializable { private List parameterProviders; private List flowRegistryClients; private List flowAnalysisRules; + private List connectors; @Schema(description = "Public interfaces defined in this bundle") public List getApis() { @@ -97,4 +98,13 @@ public void setFlowRegistryClients(List flowRegist this.flowRegistryClients = flowRegistryClients; } + @Schema(description = "Connectors provided in this bundle") + public List getConnectors() { + return (connectors != null ? Collections.unmodifiableList(connectors) : null); + } + + public void setConnectors(List connectors) { + this.connectors = connectors; + } + } diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStep.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStep.java new file mode 100644 index 000000000000..88d76b69af74 --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStep.java @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +import io.swagger.v3.oas.annotations.media.Schema; + +import java.io.Serializable; +import java.util.List; + +/** + * Represents a configuration step for a Connector. + */ +public class ConfigurationStep implements Serializable { + private static final long serialVersionUID = 1L; + + private String name; + private String description; + private boolean documented; + private List stepDependencies; + private List propertyGroups; + + @Schema(description = "The name of the configuration step") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Schema(description = "The description of the configuration step") + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Schema(description = "Whether this configuration step has additional documentation") + public boolean isDocumented() { + return documented; + } + + public void setDocumented(boolean documented) { + this.documented = documented; + } + + @Schema(description = "The dependencies that this step has on other steps") + public List getStepDependencies() { + return stepDependencies; + } + + public void setStepDependencies(List stepDependencies) { + this.stepDependencies = stepDependencies; + } + + @Schema(description = "The property groups in this configuration step") + public List getPropertyGroups() { + return propertyGroups; + } + + public void setPropertyGroups(List propertyGroups) { + this.propertyGroups = propertyGroups; + } +} + diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStepDependency.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStepDependency.java new file mode 100644 index 000000000000..1f1bc2a24d9e --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConfigurationStepDependency.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +import io.swagger.v3.oas.annotations.media.Schema; + +import java.io.Serializable; +import java.util.List; + +/** + * Represents a dependency that a configuration step has on another step's property. + */ +public class ConfigurationStepDependency implements Serializable { + private static final long serialVersionUID = 1L; + + private String stepName; + private String propertyName; + private List dependentValues; + + @Schema(description = "The name of the step that this step depends on") + public String getStepName() { + return stepName; + } + + public void setStepName(String stepName) { + this.stepName = stepName; + } + + @Schema(description = "The name of the property within the step that this step depends on") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(String propertyName) { + this.propertyName = propertyName; + } + + @Schema(description = "The values of the dependent property that enable this step") + public List getDependentValues() { + return dependentValues; + } + + public void setDependentValues(List dependentValues) { + this.dependentValues = dependentValues; + } +} + diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorDefinition.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorDefinition.java new file mode 100644 index 000000000000..9ac1c85eea3f --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorDefinition.java @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +import io.swagger.v3.oas.annotations.media.Schema; + +import java.util.Collections; +import java.util.List; + +/** + * Definition of a Connector component. + */ +public class ConnectorDefinition extends ExtensionComponent { + private static final long serialVersionUID = 1L; + + private List configurationSteps; + + @Schema(description = "The configuration steps for this connector") + public List getConfigurationSteps() { + return (configurationSteps != null ? Collections.unmodifiableList(configurationSteps) : null); + } + + public void setConfigurationSteps(List configurationSteps) { + this.configurationSteps = configurationSteps; + } +} + diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDependency.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDependency.java new file mode 100644 index 000000000000..b01b4143e0f6 --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDependency.java @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +import io.swagger.v3.oas.annotations.media.Schema; + +import java.io.Serializable; +import java.util.List; + +/** + * Represents a dependency that a connector property has on another property. + */ +public class ConnectorPropertyDependency implements Serializable { + private static final long serialVersionUID = 1L; + + private String propertyName; + private List dependentValues; + + @Schema(description = "The name of the property that this property depends on") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(String propertyName) { + this.propertyName = propertyName; + } + + @Schema(description = "The values of the dependent property that enable this property") + public List getDependentValues() { + return dependentValues; + } + + public void setDependentValues(List dependentValues) { + this.dependentValues = dependentValues; + } +} + diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDescriptor.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDescriptor.java new file mode 100644 index 000000000000..fc9b4519e4e7 --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyDescriptor.java @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +import io.swagger.v3.oas.annotations.media.Schema; + +import java.io.Serializable; +import java.util.List; + +/** + * Represents a property descriptor for a Connector. + */ +public class ConnectorPropertyDescriptor implements Serializable { + private static final long serialVersionUID = 1L; + + private String name; + private String description; + private String defaultValue; + private boolean required; + private ConnectorPropertyType propertyType; + private boolean allowableValuesFetchable; + private List allowableValues; + private List dependencies; + + @Schema(description = "The name of the property") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Schema(description = "The description of the property") + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Schema(description = "The default value of the property") + public String getDefaultValue() { + return defaultValue; + } + + public void setDefaultValue(String defaultValue) { + this.defaultValue = defaultValue; + } + + @Schema(description = "Whether or not the property is required") + public boolean isRequired() { + return required; + } + + public void setRequired(boolean required) { + this.required = required; + } + + @Schema(description = "The type of the property") + public ConnectorPropertyType getPropertyType() { + return propertyType; + } + + public void setPropertyType(ConnectorPropertyType propertyType) { + this.propertyType = propertyType; + } + + @Schema(description = "Whether or not the allowable values can be fetched dynamically") + public boolean isAllowableValuesFetchable() { + return allowableValuesFetchable; + } + + public void setAllowableValuesFetchable(boolean allowableValuesFetchable) { + this.allowableValuesFetchable = allowableValuesFetchable; + } + + @Schema(description = "The allowable values for this property") + public List getAllowableValues() { + return allowableValues; + } + + public void setAllowableValues(List allowableValues) { + this.allowableValues = allowableValues; + } + + @Schema(description = "The properties that this property depends on") + public List getDependencies() { + return dependencies; + } + + public void setDependencies(List dependencies) { + this.dependencies = dependencies; + } +} + diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyGroup.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyGroup.java new file mode 100644 index 000000000000..8eaab4ec9745 --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyGroup.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +import io.swagger.v3.oas.annotations.media.Schema; + +import java.io.Serializable; +import java.util.List; + +/** + * Represents a group of properties within a configuration step. + */ +public class ConnectorPropertyGroup implements Serializable { + private static final long serialVersionUID = 1L; + + private String name; + private String description; + private List properties; + + @Schema(description = "The name of the property group") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Schema(description = "The description of the property group") + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Schema(description = "The properties in this group") + public List getProperties() { + return properties; + } + + public void setProperties(List properties) { + this.properties = properties; + } +} + diff --git a/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyType.java b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyType.java new file mode 100644 index 000000000000..906673ec9d1a --- /dev/null +++ b/c2/c2-protocol/c2-protocol-component-api/src/main/java/org/apache/nifi/c2/protocol/component/api/ConnectorPropertyType.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.c2.protocol.component.api; + +/** + * Possible types for connector properties. + */ +public enum ConnectorPropertyType { + + STRING, + + INTEGER, + + BOOLEAN, + + FLOAT, + + DOUBLE, + + STRING_LIST, + + SECRET, + + ASSET, + + ASSET_LIST; + +} + diff --git a/checkstyle.xml b/checkstyle.xml index dc5e8a81d6a5..cd784213ed38 100644 --- a/checkstyle.xml +++ b/checkstyle.xml @@ -25,10 +25,6 @@ - - - - @@ -154,8 +150,5 @@ - - - diff --git a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java index ed0434949cd3..7998a92065ab 100644 --- a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java +++ b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/AgentPropertyValidationContext.java @@ -109,4 +109,9 @@ public boolean isDependencySatisfied(PropertyDescriptor propertyDescriptor, Func public PropertyValue getProperty(PropertyDescriptor descriptor) { throw new UnsupportedOperationException(); } + + @Override + public String evaluateParameters(final String value) { + throw new UnsupportedOperationException(); + } } diff --git a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java index ad23f6204afb..18252878cffa 100644 --- a/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java +++ b/minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/DefaultFlowStateStrategyTest.java @@ -31,7 +31,6 @@ import java.util.Set; import java.util.concurrent.CompletableFuture; -import java.util.concurrent.Future; import static org.apache.nifi.c2.protocol.api.C2OperationState.OperationState.FULLY_APPLIED; import static org.apache.nifi.c2.protocol.api.C2OperationState.OperationState.NOT_APPLIED; @@ -111,13 +110,13 @@ public void testStopFullyApplied() { ProcessGroup rootProcessGroup = mock(ProcessGroup.class); RemoteProcessGroup remoteProcessGroup = mock(RemoteProcessGroup.class); Set remoteProcessGroups = Set.of(remoteProcessGroup); - CompletableFuture rootProcessGroupStopFuture = mock(CompletableFuture.class); - Future remoteProcessGroupStopFuture = mock(Future.class); + CompletableFuture rootProcessGroupStopFuture = CompletableFuture.completedFuture(null); + CompletableFuture remoteProcessGroupStopFuture = CompletableFuture.completedFuture(null); ProcessGroup nestedProcessGroup = mock(ProcessGroup.class); RemoteProcessGroup nestedRemoteProcessGroup = mock(RemoteProcessGroup.class); Set nestedRemoteProcessGroups = Set.of(nestedRemoteProcessGroup); - CompletableFuture nestedProcessGroupStopFuture = mock(CompletableFuture.class); - Future nestedRemoteProcessGroupStopFuture = mock(Future.class); + CompletableFuture nestedProcessGroupStopFuture = CompletableFuture.completedFuture(null); + CompletableFuture nestedRemoteProcessGroupStopFuture = CompletableFuture.completedFuture(null); when(flowManager.getRootGroup()).thenReturn(rootProcessGroup); when(rootProcessGroup.getRemoteProcessGroups()).thenReturn(remoteProcessGroups); diff --git a/nifi-assembly/pom.xml b/nifi-assembly/pom.xml index 8308a723bdf6..def628fa54e5 100644 --- a/nifi-assembly/pom.xml +++ b/nifi-assembly/pom.xml @@ -915,6 +915,12 @@ language governing permissions and limitations under the License. --> 2.9.0-SNAPSHOT nar + + org.apache.nifi + nifi-kafka-to-s3-nar + 2.9.0-SNAPSHOT + nar + org.aspectj diff --git a/nifi-commons/nifi-connector-utils/pom.xml b/nifi-commons/nifi-connector-utils/pom.xml new file mode 100644 index 000000000000..c44893483cdd --- /dev/null +++ b/nifi-commons/nifi-connector-utils/pom.xml @@ -0,0 +1,42 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-commons + 2.9.0-SNAPSHOT + + + nifi-connector-utils + nifi-connector-utils + + + + org.slf4j + slf4j-api + + + org.apache.nifi + nifi-api + + + com.fasterxml.jackson.core + jackson-databind + + + + diff --git a/nifi-commons/nifi-connector-utils/src/main/java/org/apache/nifi/components/connector/util/VersionedFlowUtils.java b/nifi-commons/nifi-connector-utils/src/main/java/org/apache/nifi/components/connector/util/VersionedFlowUtils.java new file mode 100644 index 000000000000..739280d16fa1 --- /dev/null +++ b/nifi-commons/nifi-connector-utils/src/main/java/org/apache/nifi/components/connector/util/VersionedFlowUtils.java @@ -0,0 +1,592 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.util; + +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.nifi.components.connector.ComponentBundleLookup; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.ComponentType; +import org.apache.nifi.flow.ConnectableComponent; +import org.apache.nifi.flow.ConnectableComponentType; +import org.apache.nifi.flow.PortType; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; +import java.util.function.Predicate; + +public class VersionedFlowUtils { + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); + + public static VersionedExternalFlow loadFlowFromResource(final String resourceName) { + try (final InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream(resourceName)) { + if (in == null) { + throw new IllegalArgumentException("Resource not found: " + resourceName); + } + + return OBJECT_MAPPER.readValue(in, VersionedExternalFlow.class); + } catch (final Exception e) { + throw new IllegalStateException("Unable to load resource: " + resourceName, e); + } + } + + public static Optional findProcessor(final VersionedProcessGroup group, final Predicate predicate) { + final List processors = findProcessors(group, predicate); + if (processors.size() == 1) { + return Optional.of(processors.getFirst()); + } + return Optional.empty(); + } + + public static List findProcessors(final VersionedProcessGroup group, final Predicate predicate) { + final List processors = new ArrayList<>(); + findProcessors(group, predicate, processors); + return processors; + } + + private static void findProcessors(final VersionedProcessGroup group, final Predicate predicate, final List processors) { + for (final VersionedProcessor processor : group.getProcessors()) { + if (predicate.test(processor)) { + processors.add(processor); + } + } + + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + findProcessors(childGroup, predicate, processors); + } + } + + public static ConnectableComponent createConnectableComponent(final VersionedProcessor processor) { + final ConnectableComponent component = new ConnectableComponent(); + component.setId(processor.getIdentifier()); + component.setName(processor.getName()); + component.setType(ConnectableComponentType.PROCESSOR); + component.setGroupId(processor.getGroupIdentifier()); + return component; + } + + public static ConnectableComponent createConnectableComponent(final VersionedPort port) { + final ConnectableComponent component = new ConnectableComponent(); + component.setId(port.getIdentifier()); + component.setName(port.getName()); + component.setType(port.getComponentType() == ComponentType.INPUT_PORT ? ConnectableComponentType.INPUT_PORT : ConnectableComponentType.OUTPUT_PORT); + component.setGroupId(port.getGroupIdentifier()); + return component; + } + + public static VersionedConnection addConnection(final VersionedProcessGroup group, final ConnectableComponent source, final ConnectableComponent destination, + final Set relationships) { + final VersionedConnection connection = new VersionedConnection(); + connection.setSource(source); + connection.setDestination(destination); + connection.setSelectedRelationships(relationships); + connection.setBends(List.of()); + connection.setLabelIndex(0); + connection.setzIndex(0L); + connection.setGroupIdentifier(group.getIdentifier()); + connection.setLoadBalanceStrategy("DO_NOT_LOAD_BALANCE"); + connection.setBackPressureDataSizeThreshold("1 GB"); + connection.setBackPressureObjectThreshold(10000L); + connection.setFlowFileExpiration("0 sec"); + connection.setPrioritizers(new ArrayList<>()); + connection.setComponentType(ComponentType.CONNECTION); + + Set connections = group.getConnections(); + if (connections == null) { + connections = new HashSet<>(); + group.setConnections(connections); + } + connections.add(connection); + + final String uuid = generateDeterministicUuid(group, ComponentType.CONNECTION); + connection.setIdentifier(uuid); + return connection; + } + + public static List findOutboundConnections(final VersionedProcessGroup group, final VersionedProcessor processor) { + final VersionedProcessGroup processorGroup = findGroupForProcessor(group, processor); + if (processorGroup == null) { + return List.of(); + } + + final List outboundConnections = new ArrayList<>(); + final Set connections = processorGroup.getConnections(); + if (connections == null) { + return outboundConnections; + } + + for (final VersionedConnection connection : connections) { + final ConnectableComponent source = connection.getSource(); + if (Objects.equals(source.getId(), processor.getIdentifier()) && source.getType() == ConnectableComponentType.PROCESSOR) { + outboundConnections.add(connection); + } + } + + return outboundConnections; + } + + public static VersionedProcessGroup findGroupForProcessor(final VersionedProcessGroup rootGroup, final VersionedProcessor processor) { + if (rootGroup.getProcessors().contains(processor)) { + return rootGroup; + } + + for (final VersionedProcessGroup childGroup : rootGroup.getProcessGroups()) { + final VersionedProcessGroup foundGroup = findGroupForProcessor(childGroup, processor); + if (foundGroup != null) { + return foundGroup; + } + } + + return null; + } + + public static String generateDeterministicUuid(final VersionedProcessGroup group, final ComponentType componentType) { + final int componentCount = getComponentCount(group, componentType); + final String uuidSeed = "%s-%s-%d".formatted(group.getIdentifier(), componentType.name(), componentCount); + return UUID.nameUUIDFromBytes(uuidSeed.getBytes(StandardCharsets.UTF_8)).toString(); + } + + private static int getComponentCount(final VersionedProcessGroup group, final ComponentType componentType) { + final Collection components = switch (componentType) { + case PROCESSOR -> group.getProcessors(); + case INPUT_PORT -> group.getInputPorts(); + case OUTPUT_PORT -> group.getOutputPorts(); + case CONNECTION -> group.getConnections(); + case FUNNEL -> group.getFunnels(); + case LABEL -> group.getLabels(); + case PROCESS_GROUP -> group.getProcessGroups(); + case CONTROLLER_SERVICE -> group.getControllerServices(); + default -> List.of(); + }; + + return components == null ? 0 : components.size(); + } + + public static VersionedProcessor addProcessor(final VersionedProcessGroup group, final String processorType, final Bundle bundle, final String name, final Position position) { + final VersionedProcessor processor = new VersionedProcessor(); + + // Generate deterministic UUID based on group and component type + processor.setIdentifier(generateDeterministicUuid(group, ComponentType.PROCESSOR)); + + processor.setName(name); + processor.setType(processorType); + processor.setPosition(position); + processor.setBundle(bundle); + + // Set default processor configuration + processor.setProperties(new HashMap<>()); + processor.setPropertyDescriptors(new HashMap<>()); + processor.setStyle(new HashMap<>()); + processor.setSchedulingPeriod("0 sec"); + processor.setSchedulingStrategy("TIMER_DRIVEN"); + processor.setExecutionNode("ALL"); + processor.setPenaltyDuration("30 sec"); + processor.setYieldDuration("1 sec"); + processor.setBulletinLevel("WARN"); + processor.setRunDurationMillis(25L); + processor.setConcurrentlySchedulableTaskCount(1); + processor.setAutoTerminatedRelationships(new HashSet<>()); + processor.setScheduledState(ScheduledState.ENABLED); + processor.setRetryCount(10); + processor.setRetriedRelationships(new HashSet<>()); + processor.setBackoffMechanism("PENALIZE_FLOWFILE"); + processor.setMaxBackoffPeriod("10 mins"); + processor.setComponentType(ComponentType.PROCESSOR); + processor.setGroupIdentifier(group.getIdentifier()); + + group.getProcessors().add(processor); + return processor; + } + + public static VersionedControllerService addControllerService(final VersionedProcessGroup group, final String serviceType, final Bundle bundle, final String name) { + final VersionedControllerService controllerService = new VersionedControllerService(); + + // Generate deterministic UUID based on group and component type + controllerService.setIdentifier(generateDeterministicUuid(group, ComponentType.CONTROLLER_SERVICE)); + + controllerService.setName(name); + controllerService.setType(serviceType); + controllerService.setBundle(bundle); + controllerService.setComponentType(ComponentType.CONTROLLER_SERVICE); + + // Set default controller service configuration + controllerService.setProperties(new HashMap<>()); + controllerService.setPropertyDescriptors(new HashMap<>()); + controllerService.setControllerServiceApis(new ArrayList<>()); + controllerService.setAnnotationData(null); + controllerService.setScheduledState(ScheduledState.DISABLED); + controllerService.setBulletinLevel("WARN"); + controllerService.setComments(null); + controllerService.setGroupIdentifier(group.getIdentifier()); + + // Initialize controller services collection if it doesn't exist + Set controllerServices = group.getControllerServices(); + if (controllerServices == null) { + controllerServices = new HashSet<>(); + group.setControllerServices(controllerServices); + } + controllerServices.add(controllerService); + + return controllerService; + } + + public static VersionedProcessGroup createProcessGroup(final String identifier, final String name) { + final VersionedProcessGroup group = new VersionedProcessGroup(); + group.setIdentifier(identifier); + group.setName(name); + group.setProcessors(new HashSet<>()); + group.setProcessGroups(new HashSet<>()); + group.setConnections(new HashSet<>()); + group.setControllerServices(new HashSet<>()); + group.setInputPorts(new HashSet<>()); + group.setOutputPorts(new HashSet<>()); + group.setFunnels(new HashSet<>()); + group.setLabels(new HashSet<>()); + group.setComponentType(ComponentType.PROCESS_GROUP); + return group; + } + + public static VersionedPort addInputPort(final VersionedProcessGroup group, final String name, final Position position) { + return addPort(group, name, position, PortType.INPUT_PORT); + } + + public static VersionedPort addOutputPort(final VersionedProcessGroup group, final String name, final Position position) { + return addPort(group, name, position, PortType.OUTPUT_PORT); + } + + private static VersionedPort addPort(final VersionedProcessGroup group, final String name, final Position position, final PortType portType) { + final boolean isInput = portType == PortType.INPUT_PORT; + final ComponentType componentType = isInput ? ComponentType.INPUT_PORT : ComponentType.OUTPUT_PORT; + + final VersionedPort port = new VersionedPort(); + port.setIdentifier(generateDeterministicUuid(group, componentType)); + port.setName(name); + port.setPosition(position); + port.setType(portType); + port.setComponentType(componentType); + port.setScheduledState(ScheduledState.ENABLED); + port.setConcurrentlySchedulableTaskCount(1); + port.setAllowRemoteAccess(false); + port.setGroupIdentifier(group.getIdentifier()); + + if (isInput) { + Set inputPorts = group.getInputPorts(); + if (inputPorts == null) { + inputPorts = new HashSet<>(); + group.setInputPorts(inputPorts); + } + inputPorts.add(port); + } else { + Set outputPorts = group.getOutputPorts(); + if (outputPorts == null) { + outputPorts = new HashSet<>(); + group.setOutputPorts(outputPorts); + } + outputPorts.add(port); + } + + return port; + } + + public static Set getReferencedControllerServices(final VersionedProcessGroup group) { + final Set referencedServices = new HashSet<>(); + collectReferencedControllerServices(group, referencedServices); + return referencedServices; + } + + private static void collectReferencedControllerServices(final VersionedProcessGroup group, final Set referencedServices) { + final Map serviceMap = new HashMap<>(); + for (final VersionedControllerService service : group.getControllerServices()) { + serviceMap.put(service.getIdentifier(), service); + } + + for (final VersionedProcessor processor : group.getProcessors()) { + for (final String propertyValue : processor.getProperties().values()) { + final VersionedControllerService referencedService = serviceMap.get(propertyValue); + if (referencedService != null) { + referencedServices.add(referencedService); + } + } + } + + while (true) { + final Set newlyAddedServices = new HashSet<>(); + + for (final VersionedControllerService service : referencedServices) { + for (final String propertyValue : service.getProperties().values()) { + final VersionedControllerService referencedService = serviceMap.get(propertyValue); + if (referencedService != null && !referencedServices.contains(referencedService)) { + newlyAddedServices.add(referencedService); + } + } + } + + referencedServices.addAll(newlyAddedServices); + if (newlyAddedServices.isEmpty()) { + break; + } + } + + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + collectReferencedControllerServices(childGroup, referencedServices); + } + } + + /** + * Returns the set of controller services that are transitively referenced by the given processor. + * This includes any services directly referenced by the processor's properties, as well as any services + * that those services reference, and so on. Only services that are accessible to the processor are considered, + * meaning services in the processor's own group and its ancestor groups. + * + * @param rootGroup the root process group to search for controller services + * @param processor the processor whose referenced services should be found + * @return the set of transitively referenced controller services + */ + public static Set getReferencedControllerServices(final VersionedProcessGroup rootGroup, final VersionedProcessor processor) { + return findTransitivelyReferencedServices(rootGroup, processor.getGroupIdentifier(), processor.getProperties()); + } + + /** + * Returns the set of controller services that are transitively referenced by the given controller service. + * This includes any services directly referenced by the service's properties, as well as any services + * that those services reference, and so on. Only services that are accessible to the given service are considered, + * meaning services in the service's own group and its ancestor groups. + * + * @param rootGroup the root process group to search for controller services + * @param controllerService the controller service whose referenced services should be found + * @return the set of transitively referenced controller services + */ + public static Set getReferencedControllerServices(final VersionedProcessGroup rootGroup, final VersionedControllerService controllerService) { + return findTransitivelyReferencedServices(rootGroup, controllerService.getGroupIdentifier(), controllerService.getProperties()); + } + + private static Set findTransitivelyReferencedServices(final VersionedProcessGroup rootGroup, final String componentGroupId, + final Map properties) { + final Map serviceMap = new HashMap<>(); + collectAccessibleControllerServices(rootGroup, componentGroupId, serviceMap); + + final Set referencedServices = new HashSet<>(); + for (final String propertyValue : properties.values()) { + final VersionedControllerService referencedService = serviceMap.get(propertyValue); + if (referencedService != null) { + referencedServices.add(referencedService); + } + } + + resolveTransitiveServiceReferences(referencedServices, serviceMap); + return referencedServices; + } + + /** + * Collects controller services that are accessible from the given target group. In NiFi, a component can reference + * controller services in its own group or any ancestor group. This method traverses from the root group down to the + * target group, collecting services from each group along the path. + * + * @param group the current group being examined + * @param targetGroupId the identifier of the group whose accessible services should be collected + * @param serviceMap the map to populate with accessible service identifiers and their corresponding services + * @return true if the target group was found at or beneath this group, false otherwise + */ + private static boolean collectAccessibleControllerServices(final VersionedProcessGroup group, final String targetGroupId, + final Map serviceMap) { + final boolean isTarget = group.getIdentifier().equals(targetGroupId); + + boolean foundInChild = false; + if (!isTarget) { + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + if (collectAccessibleControllerServices(childGroup, targetGroupId, serviceMap)) { + foundInChild = true; + break; + } + } + } + + if (isTarget || foundInChild) { + for (final VersionedControllerService service : group.getControllerServices()) { + serviceMap.put(service.getIdentifier(), service); + } + return true; + } + + return false; + } + + private static void resolveTransitiveServiceReferences(final Set referencedServices, final Map serviceMap) { + while (true) { + final Set newlyAddedServices = new HashSet<>(); + + for (final VersionedControllerService service : referencedServices) { + for (final String propertyValue : service.getProperties().values()) { + final VersionedControllerService referencedService = serviceMap.get(propertyValue); + if (referencedService != null && !referencedServices.contains(referencedService)) { + newlyAddedServices.add(referencedService); + } + } + } + + referencedServices.addAll(newlyAddedServices); + if (newlyAddedServices.isEmpty()) { + break; + } + } + } + + public static void removeControllerServiceReferences(final VersionedProcessGroup processGroup, final String serviceIdentifier) { + for (final VersionedProcessor processor : processGroup.getProcessors()) { + removeValuesFromMap(processor.getProperties(), serviceIdentifier); + } + + for (final VersionedControllerService service : processGroup.getControllerServices()) { + removeValuesFromMap(service.getProperties(), serviceIdentifier); + } + + for (final VersionedProcessGroup childGroup : processGroup.getProcessGroups()) { + removeControllerServiceReferences(childGroup, serviceIdentifier); + } + } + + private static void removeValuesFromMap(final Map properties, final String valueToRemove) { + final List keysToRemove = new ArrayList<>(); + for (final Map.Entry entry : properties.entrySet()) { + if (Objects.equals(entry.getValue(), valueToRemove)) { + keysToRemove.add(entry.getKey()); + } + } + + keysToRemove.forEach(properties::remove); + } + + public static void setParameterValue(final VersionedExternalFlow externalFlow, final String parameterName, final String parameterValue) { + for (final VersionedParameterContext context : externalFlow.getParameterContexts().values()) { + setParameterValue(context, parameterName, parameterValue); + } + } + + public static void setParameterValue(final VersionedParameterContext parameterContext, final String parameterName, final String parameterValue) { + final Set parameters = parameterContext.getParameters(); + for (final VersionedParameter parameter : parameters) { + if (parameter.getName().equals(parameterName)) { + parameter.setValue(parameterValue); + } + } + } + + public static void setParameterValues(final VersionedExternalFlow externalFlow, final Map parameterValues) { + for (final Map.Entry entry : parameterValues.entrySet()) { + setParameterValue(externalFlow, entry.getKey(), entry.getValue()); + } + } + + public static void setParameterValues(final VersionedParameterContext parameterContext, final Map parameterValues) { + for (final Map.Entry entry : parameterValues.entrySet()) { + setParameterValue(parameterContext, entry.getKey(), entry.getValue()); + } + } + + public static void removeUnreferencedControllerServices(final VersionedProcessGroup processGroup) { + final Set referencedServices = getReferencedControllerServices(processGroup); + final Set referencedServiceIds = new HashSet<>(); + for (final VersionedControllerService service : referencedServices) { + referencedServiceIds.add(service.getIdentifier()); + } + + removeUnreferencedControllerServices(processGroup, referencedServiceIds); + } + + private static void removeUnreferencedControllerServices(final VersionedProcessGroup processGroup, final Set referencedServiceIds) { + processGroup.getControllerServices().removeIf(service -> !referencedServiceIds.contains(service.getIdentifier())); + + for (final VersionedProcessGroup childGroup : processGroup.getProcessGroups()) { + removeUnreferencedControllerServices(childGroup, referencedServiceIds); + } + } + + /** + * Updates all processors and controller services in the given process group (and its child groups) + * to use the latest available bundle version. See {@link ComponentBundleLookup#getLatestBundle(String)} for details on how + * version comparison is performed. + * + * @param processGroup the process group containing components to update + * @param componentBundleLookup the lookup used to find available bundles for each component type + */ + public static void updateToLatestBundles(final VersionedProcessGroup processGroup, final ComponentBundleLookup componentBundleLookup) { + for (final VersionedProcessor processor : processGroup.getProcessors()) { + updateToLatestBundle(processor, componentBundleLookup); + } + + for (final VersionedControllerService service : processGroup.getControllerServices()) { + updateToLatestBundle(service, componentBundleLookup); + } + + for (final VersionedProcessGroup childGroup : processGroup.getProcessGroups()) { + updateToLatestBundles(childGroup, componentBundleLookup); + } + } + + /** + * Updates the given processor to use the latest available bundle version. + * If no bundle is available, the processor's bundle is left unchanged. + * See {@link ComponentBundleLookup#getLatestBundle(String)} for details on how version comparison is performed. + * + * @param processor the processor to update + * @param componentBundleLookup the lookup used to find available bundles for the processor type + * @return true if the bundle was updated, false if no bundle was available + */ + public static boolean updateToLatestBundle(final VersionedProcessor processor, final ComponentBundleLookup componentBundleLookup) { + final Optional latestBundle = componentBundleLookup.getLatestBundle(processor.getType()); + latestBundle.ifPresent(processor::setBundle); + return latestBundle.isPresent(); + } + + /** + * Updates the given controller service to use the latest available bundle version. + * If no bundle is available, the service's bundle is left unchanged. + * See {@link ComponentBundleLookup#getLatestBundle(String)} for details on how version comparison is performed. + * + * @param service the controller service to update + * @param componentBundleLookup the lookup used to find available bundles for the service type + * @return true if the bundle was updated, false if no bundle was available + */ + public static boolean updateToLatestBundle(final VersionedControllerService service, final ComponentBundleLookup componentBundleLookup) { + final Optional latestBundle = componentBundleLookup.getLatestBundle(service.getType()); + latestBundle.ifPresent(service::setBundle); + return latestBundle.isPresent(); + } + +} diff --git a/nifi-commons/nifi-connector-utils/src/test/java/org/apache/nifi/components/connector/util/TestVersionedFlowUtils.java b/nifi-commons/nifi-connector-utils/src/test/java/org/apache/nifi/components/connector/util/TestVersionedFlowUtils.java new file mode 100644 index 000000000000..a7534d9088f5 --- /dev/null +++ b/nifi-commons/nifi-connector-utils/src/test/java/org/apache/nifi/components/connector/util/TestVersionedFlowUtils.java @@ -0,0 +1,285 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.util; + +import org.apache.nifi.components.connector.ComponentBundleLookup; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import java.util.Optional; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestVersionedFlowUtils { + + @Nested + class UpdateToLatestBundles { + private static final String PROCESSOR_TYPE = "org.apache.nifi.processors.TestProcessor"; + private static final String SERVICE_TYPE = "org.apache.nifi.services.TestService"; + + @Test + void testLookupReturnsNewerBundle() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0)); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "3.0.0"))); + + VersionedFlowUtils.updateToLatestBundles(group, lookup); + + assertEquals("3.0.0", processor.getBundle().getVersion()); + } + + @Test + void testLookupReturnsLatestBundle() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0)); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "4.0.0"))); + + VersionedFlowUtils.updateToLatestBundles(group, lookup); + + assertEquals("4.0.0", processor.getBundle().getVersion()); + } + + @Test + void testControllerServiceUpdatedToNewerBundle() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedControllerService service = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, new Bundle("group", "artifact", "1.5.0"), "Test Service"); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(SERVICE_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "2.5.0"))); + + VersionedFlowUtils.updateToLatestBundles(group, lookup); + + assertEquals("2.5.0", service.getBundle().getVersion()); + } + + @Test + void testNestedProcessorUpdatedToNewerBundle() { + final VersionedProcessGroup rootGroup = createProcessGroup(); + final VersionedProcessGroup childGroup = createChildProcessGroup(rootGroup, "child-group-id"); + final VersionedProcessor nestedProcessor = VersionedFlowUtils.addProcessor(childGroup, PROCESSOR_TYPE, new Bundle("group", "artifact", "1.0.0"), "Nested Processor", new Position(0, 0)); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "5.0.0"))); + + VersionedFlowUtils.updateToLatestBundles(rootGroup, lookup); + + assertEquals("5.0.0", nestedProcessor.getBundle().getVersion()); + } + + @Test + void testEmptyOptionalDoesNotChangeBundle() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0)); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.empty()); + + VersionedFlowUtils.updateToLatestBundles(group, lookup); + + assertEquals("2.0.0", processor.getBundle().getVersion()); + } + + @Test + void testUpdateToLatestBundleReturnsTrueWhenUpdated() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0)); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.of(new Bundle("group", "artifact", "3.0.0"))); + + final boolean updated = VersionedFlowUtils.updateToLatestBundle(processor, lookup); + + assertTrue(updated); + assertEquals("3.0.0", processor.getBundle().getVersion()); + } + + @Test + void testUpdateToLatestBundleReturnsFalseWhenNotUpdated() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, new Bundle("group", "artifact", "2.0.0"), "Test Processor", new Position(0, 0)); + + final ComponentBundleLookup lookup = mock(ComponentBundleLookup.class); + when(lookup.getLatestBundle(PROCESSOR_TYPE)).thenReturn(Optional.empty()); + + final boolean updated = VersionedFlowUtils.updateToLatestBundle(processor, lookup); + + assertFalse(updated); + assertEquals("2.0.0", processor.getBundle().getVersion()); + } + + } + + @Nested + class GetReferencedControllerServices { + private static final String PROCESSOR_TYPE = "org.apache.nifi.processors.TestProcessor"; + private static final String SERVICE_TYPE = "org.apache.nifi.services.TestService"; + private static final Bundle TEST_BUNDLE = new Bundle("group", "artifact", "1.0.0"); + + @Test + void testProcessorWithNoReferences() { + final VersionedProcessGroup group = createProcessGroup(); + VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Unreferenced Service"); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0)); + processor.getProperties().put("some-property", "not-a-service-id"); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(group, processor); + + assertTrue(result.isEmpty()); + } + + @Test + void testProcessorWithDirectReferences() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedControllerService service1 = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service 1"); + final VersionedControllerService service2 = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service 2"); + final VersionedControllerService unreferencedService = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Unreferenced Service"); + + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0)); + processor.getProperties().put("service-prop-1", service1.getIdentifier()); + processor.getProperties().put("service-prop-2", service2.getIdentifier()); + processor.getProperties().put("non-service-prop", "some-value"); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(group, processor); + + assertEquals(2, result.size()); + assertTrue(result.contains(service1)); + assertTrue(result.contains(service2)); + assertFalse(result.contains(unreferencedService)); + } + + @Test + void testProcessorWithTransitiveReferences() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedControllerService serviceC = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service C"); + final VersionedControllerService serviceB = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service B"); + serviceB.getProperties().put("nested-service", serviceC.getIdentifier()); + final VersionedControllerService serviceA = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service A"); + serviceA.getProperties().put("nested-service", serviceB.getIdentifier()); + + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0)); + processor.getProperties().put("service-prop", serviceA.getIdentifier()); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(group, processor); + + assertEquals(3, result.size()); + assertTrue(result.contains(serviceA)); + assertTrue(result.contains(serviceB)); + assertTrue(result.contains(serviceC)); + } + + @Test + void testProcessorReferencingServiceInAncestorGroup() { + final VersionedProcessGroup rootGroup = createProcessGroup(); + final VersionedControllerService parentService = VersionedFlowUtils.addControllerService(rootGroup, SERVICE_TYPE, TEST_BUNDLE, "Parent Service"); + final VersionedProcessGroup childGroup = createChildProcessGroup(rootGroup, "child-group-id"); + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(childGroup, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0)); + processor.getProperties().put("service-prop", parentService.getIdentifier()); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(rootGroup, processor); + + assertEquals(1, result.size()); + assertTrue(result.contains(parentService)); + } + + @Test + void testProcessorDoesNotFindServiceInDescendantGroup() { + final VersionedProcessGroup rootGroup = createProcessGroup(); + final VersionedProcessGroup childGroup = createChildProcessGroup(rootGroup, "child-group-id"); + final VersionedControllerService childService = VersionedFlowUtils.addControllerService(childGroup, SERVICE_TYPE, TEST_BUNDLE, "Child Service"); + + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(rootGroup, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0)); + processor.getProperties().put("service-prop", childService.getIdentifier()); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(rootGroup, processor); + + assertTrue(result.isEmpty()); + } + + @Test + void testProcessorWithCircularServiceReferences() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedControllerService serviceA = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service A"); + final VersionedControllerService serviceB = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service B"); + serviceA.getProperties().put("nested-service", serviceB.getIdentifier()); + serviceB.getProperties().put("nested-service", serviceA.getIdentifier()); + + final VersionedProcessor processor = VersionedFlowUtils.addProcessor(group, PROCESSOR_TYPE, TEST_BUNDLE, "Processor", new Position(0, 0)); + processor.getProperties().put("service-prop", serviceA.getIdentifier()); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(group, processor); + + assertEquals(2, result.size()); + assertTrue(result.contains(serviceA)); + assertTrue(result.contains(serviceB)); + } + + @Test + void testControllerServiceWithNoReferences() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedControllerService service = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service"); + service.getProperties().put("some-property", "not-a-service-id"); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(group, service); + + assertTrue(result.isEmpty()); + } + + @Test + void testControllerServiceWithDirectAndTransitiveReferences() { + final VersionedProcessGroup group = createProcessGroup(); + final VersionedControllerService serviceC = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service C"); + final VersionedControllerService serviceB = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service B"); + serviceB.getProperties().put("nested-service", serviceC.getIdentifier()); + final VersionedControllerService serviceA = VersionedFlowUtils.addControllerService(group, SERVICE_TYPE, TEST_BUNDLE, "Service A"); + serviceA.getProperties().put("nested-service", serviceB.getIdentifier()); + + final Set result = VersionedFlowUtils.getReferencedControllerServices(group, serviceA); + + assertEquals(2, result.size()); + assertTrue(result.contains(serviceB)); + assertTrue(result.contains(serviceC)); + assertFalse(result.contains(serviceA)); + } + } + + private static VersionedProcessGroup createProcessGroup() { + return VersionedFlowUtils.createProcessGroup("test-group-id", "Test Process Group"); + } + + private static VersionedProcessGroup createChildProcessGroup(final VersionedProcessGroup parent, final String identifier) { + final VersionedProcessGroup childGroup = VersionedFlowUtils.createProcessGroup(identifier, "Child Process Group"); + childGroup.setGroupIdentifier(parent.getIdentifier()); + parent.getProcessGroups().add(childGroup); + return childGroup; + } + +} diff --git a/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java b/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java index 0d6a8412e658..eef06942f013 100644 --- a/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java +++ b/nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java @@ -124,6 +124,9 @@ public class NiFiProperties extends ApplicationProperties { public static final String ASSET_MANAGER_IMPLEMENTATION = "nifi.asset.manager.implementation"; public static final String ASSET_MANAGER_PREFIX = "nifi.asset.manager.properties."; + public static final String CONNECTOR_ASSET_MANAGER_IMPLEMENTATION = "nifi.connector.asset.manager.implementation"; + public static final String CONNECTOR_ASSET_MANAGER_PREFIX = "nifi.connector.asset.manager.properties."; + // status repository properties public static final String COMPONENT_STATUS_REPOSITORY_IMPLEMENTATION = "nifi.components.status.repository.implementation"; public static final String COMPONENT_STATUS_SNAPSHOT_FREQUENCY = "nifi.components.status.snapshot.frequency"; @@ -136,6 +139,16 @@ public class NiFiProperties extends ApplicationProperties { public static final String STATUS_REPOSITORY_QUESTDB_PERSIST_BATCH_SIZE = "nifi.status.repository.questdb.persist.batchsize"; public static final String STATUS_REPOSITORY_QUESTDB_PERSIST_FREQUENCY = "nifi.status.repository.questdb.persist.frequency"; + // Connector Repository properties + public static final String CONNECTOR_REPOSITORY_IMPLEMENTATION = "nifi.components.connectors.repository.implementation"; + + // Connector Configuration Provider properties + public static final String CONNECTOR_CONFIGURATION_PROVIDER_IMPLEMENTATION = "nifi.components.connectors.configuration.provider.implementation"; + public static final String CONNECTOR_CONFIGURATION_PROVIDER_PROPERTIES_PREFIX = "nifi.components.connectors.configuration.provider."; + + // Secrets Manager properties + public static final String SECRETS_MANAGER_IMPLEMENTATION = "nifi.secrets.manager.implementation"; + // security properties public static final String SECURITY_KEYSTORE = "nifi.security.keystore"; public static final String SECURITY_KEYSTORE_TYPE = "nifi.security.keystoreType"; @@ -1807,8 +1820,28 @@ public static NiFiProperties createBasicNiFiProperties(final String propertiesFi // The Properties(Properties) constructor does NOT inherit the provided values, just uses them as default values if (additionalProperties != null) { - additionalProperties.forEach(properties::put); + properties.putAll(additionalProperties); } + + return createNiFiProperties(properties); + } + + public static NiFiProperties createBasicNiFiProperties(final InputStream inputStream) { + return createBasicNiFiProperties(inputStream, null); + } + + public static NiFiProperties createBasicNiFiProperties(final InputStream inputStream, final Properties additionalProperties) { + final Properties properties = new Properties(); + readFromInputStream(inputStream, properties); + + if (additionalProperties != null) { + properties.putAll(additionalProperties); + } + + return createNiFiProperties(properties); + } + + private static NiFiProperties createNiFiProperties(final Properties properties) { return new NiFiProperties() { @Override public String getProperty(String key) { @@ -1828,9 +1861,7 @@ public int size() { } private static void readFromPropertiesFile(String propertiesFilePath, Properties properties) { - final String nfPropertiesFilePath = (propertiesFilePath == null) - ? System.getProperty(PROPERTIES_FILE_PATH) - : propertiesFilePath; + final String nfPropertiesFilePath = (propertiesFilePath == null) ? System.getProperty(PROPERTIES_FILE_PATH) : propertiesFilePath; if (nfPropertiesFilePath != null) { final File propertiesFile = new File(nfPropertiesFilePath.trim()); if (!propertiesFile.exists()) { @@ -1841,24 +1872,25 @@ private static void readFromPropertiesFile(String propertiesFilePath, Properties throw new RuntimeException("Properties file exists but cannot be read '" + propertiesFile.getAbsolutePath() + "'"); } - InputStream inStream = null; - try { - inStream = new BufferedInputStream(new FileInputStream(propertiesFile)); + + try (final InputStream fileIn = new FileInputStream(propertiesFile); + final InputStream inStream = new BufferedInputStream(fileIn)) { + properties.load(inStream); } catch (final Exception ex) { - throw new RuntimeException("Cannot load properties file due to " - + ex.getLocalizedMessage(), ex); - } finally { - if (null != inStream) { - try { - inStream.close(); - } catch (final Exception ignored) { - } - } + throw new RuntimeException("Cannot load properties file due to " + ex.getLocalizedMessage(), ex); } } } + private static void readFromInputStream(final InputStream inputStream, final Properties properties) { + try { + properties.load(inputStream); + } catch (final Exception ex) { + throw new RuntimeException("Cannot load properties file due to " + ex.getLocalizedMessage(), ex); + } + } + /** * This method is used to validate the NiFi properties when the file is loaded * for the first time. The objective is to stop NiFi startup in case a property diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java index 4c768ed76c8d..b6d7682ae71a 100644 --- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java +++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java @@ -175,14 +175,18 @@ public void update(final Collection records) { switch (updateType) { case DELETE: - recordMap.remove(recordId); + if (recordId != null) { + recordMap.remove(recordId); + } break; case SWAP_OUT: final String location = serdeFactory.getLocation(record); if (location == null) { logger.error(logMessage, recordId, UpdateType.SWAP_OUT, "Swapped Out to", "lost"); } else { - recordMap.remove(recordId); + if (recordId != null) { + recordMap.remove(recordId); + } this.swapLocations.add(location); } break; @@ -193,10 +197,30 @@ public void update(final Collection records) { } else { swapLocations.remove(swapLocation); } - recordMap.put(recordId, record); + if (recordId != null) { + recordMap.put(recordId, record); + } + break; + case SWAP_FILE_DELETED: + final String deletedSwapLocation = serdeFactory.getLocation(record); + if (deletedSwapLocation != null) { + swapLocations.remove(deletedSwapLocation); + } + break; + case SWAP_FILE_RENAMED: + final String originalLocation = serdeFactory.getOriginalLocation(record); + final String newLocation = serdeFactory.getLocation(record); + if (originalLocation != null) { + swapLocations.remove(originalLocation); + } + if (newLocation != null) { + swapLocations.add(newLocation); + } break; default: - recordMap.put(recordId, record); + if (recordId != null) { + recordMap.put(recordId, record); + } break; } } @@ -209,6 +233,9 @@ public int getRecordCount() { @Override public T lookup(final Object recordId) { + if (recordId == null) { + return null; + } return recordMap.get(recordId); } diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java index 36350d0c8e14..9b9428539e4b 100644 --- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java +++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java @@ -475,8 +475,10 @@ public JournalRecovery recoverRecords(final Map recordMap, final Set< switch (updateType) { case DELETE: { - idsRemoved.add(recordId); - transactionRecordMap.remove(recordId); + if (recordId != null) { + idsRemoved.add(recordId); + transactionRecordMap.remove(recordId); + } break; } case SWAP_IN: { @@ -486,7 +488,9 @@ public JournalRecovery recoverRecords(final Map recordMap, final Set< } else { swapLocationsRemoved.add(location); swapLocationsAdded.remove(location); - transactionRecordMap.put(recordId, record); + if (recordId != null) { + transactionRecordMap.put(recordId, record); + } } break; } @@ -497,15 +501,40 @@ public JournalRecovery recoverRecords(final Map recordMap, final Set< } else { swapLocationsRemoved.remove(location); swapLocationsAdded.add(location); - idsRemoved.add(recordId); - transactionRecordMap.remove(recordId); + if (recordId != null) { + idsRemoved.add(recordId); + transactionRecordMap.remove(recordId); + } } break; } + case SWAP_FILE_DELETED: { + final String location = serde.getLocation(record); + if (location != null) { + swapLocationsRemoved.add(location); + swapLocationsAdded.remove(location); + } + break; + } + case SWAP_FILE_RENAMED: { + final String originalLocation = serde.getOriginalLocation(record); + final String newLocation = serde.getLocation(record); + if (originalLocation != null) { + swapLocationsRemoved.add(originalLocation); + swapLocationsAdded.remove(originalLocation); + } + if (newLocation != null) { + swapLocationsAdded.add(newLocation); + swapLocationsRemoved.remove(newLocation); + } + break; + } default: { - transactionRecordMap.put(recordId, record); - idsRemoved.remove(recordId); + if (recordId != null) { + transactionRecordMap.put(recordId, record); + idsRemoved.remove(recordId); + } break; } } diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java index 356cf8457338..990484bdf967 100644 --- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java +++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDe.java @@ -136,6 +136,19 @@ default void readHeader(DataInputStream in) throws IOException { */ String getLocation(T record); + /** + * Returns the original external location of the given record; this is used when a + * swap file is renamed. For {@link UpdateType#SWAP_FILE_RENAMED} records, this + * returns the original location before the rename, while {@link #getLocation(Object)} + * returns the new location after the rename. + * + * @param record to get original location of + * @return original location, or null if not applicable + */ + default String getOriginalLocation(T record) { + return null; + } + /** * Returns the version that this SerDe will use when writing. This used used * when serializing/deserializing the edit logs so that if the version diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java index 09e6f7b50c19..26a177ece88e 100644 --- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java +++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/SerDeFactory.java @@ -57,4 +57,17 @@ public interface SerDeFactory { * @return location */ String getLocation(T record); + + /** + * Returns the original external location of the given record; this is used when a + * swap file is renamed. For {@link UpdateType#SWAP_FILE_RENAMED} records, this + * returns the original location before the rename, while {@link #getLocation(Object)} + * returns the new location after the rename. + * + * @param record to get original location of + * @return original location, or null if not applicable + */ + default String getOriginalLocation(T record) { + return null; + } } diff --git a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java index 1b039f82fcbb..0d383cbe3776 100644 --- a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java +++ b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/UpdateType.java @@ -45,5 +45,13 @@ public enum UpdateType { * Used to indicate that a Record that was previously Swapped Out is now * being Swapped In */ - SWAP_IN; + SWAP_IN, + /** + * Used to indicate that a Swap File has been deleted + */ + SWAP_FILE_DELETED, + /** + * Used to indicate that a Swap File has been renamed + */ + SWAP_FILE_RENAMED; } diff --git a/nifi-commons/pom.xml b/nifi-commons/pom.xml index eba472df95d9..488b7ef209aa 100644 --- a/nifi-commons/pom.xml +++ b/nifi-commons/pom.xml @@ -68,6 +68,7 @@ nifi-write-ahead-log nifi-xml-processing nifi-calcite-utils + nifi-connector-utils 33.5.0-jre diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-api/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-api/pom.xml new file mode 100644 index 000000000000..d43dffe3192f --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/pom.xml @@ -0,0 +1,35 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-api + jar + + + + org.apache.nifi + nifi-server-api + 2.9.0-SNAPSHOT + + + + \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorConfigVerificationResult.java b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorConfigVerificationResult.java new file mode 100644 index 000000000000..9ec3ac2878cd --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorConfigVerificationResult.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.ConfigVerificationResult; + +import java.util.List; + +public interface ConnectorConfigVerificationResult { + + List getAllResults(); + + List getFailedResults(); + + void assertNoFailures(); +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorMockServer.java b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorMockServer.java new file mode 100644 index 000000000000..ade061a9d479 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorMockServer.java @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.NiFiServer; +import org.apache.nifi.controller.ControllerService; +import org.apache.nifi.processor.Processor; + +import java.io.File; + +public interface ConnectorMockServer extends NiFiServer, ConnectorTestRunner { + + void instantiateConnector(String connectorClassName); + + void registerMockBundle(ClassLoader classLoader, File workingDirectory); + + void mockProcessor(String processorType, Class mockProcessorClass); + + void mockControllerService(String controllerServiceType, Class mockControllerServiceClass); + +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorTestRunner.java b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorTestRunner.java new file mode 100644 index 000000000000..669f7be22d26 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-api/src/main/java/org/apache/nifi/mock/connector/server/ConnectorTestRunner.java @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.io.Closeable; +import java.io.File; +import java.io.InputStream; +import java.time.Duration; +import java.util.List; +import java.util.Map; + +public interface ConnectorTestRunner extends Closeable { + String SECRET_PROVIDER_ID = "TestRunnerSecretsManager"; + String SECRET_PROVIDER_NAME = "TestRunnerSecretsManager"; + + void applyUpdate() throws FlowUpdateException; + + void configure(String stepName, StepConfiguration configuration) throws FlowUpdateException; + + void configure(String stepName, Map propertyValues) throws FlowUpdateException; + + void configure(String stepName, Map propertyValues, Map propertyReferences) throws FlowUpdateException; + + SecretReference createSecretReference(String secretName); + + ConnectorConfigVerificationResult verifyConfiguration(String stepName, Map propertyValueOverrides); + + ConnectorConfigVerificationResult verifyConfiguration(String stepName, Map propertyValueOverrides, Map referenceOverrides); + + ConnectorConfigVerificationResult verifyConfiguration(String stepName, StepConfiguration configurationOverrides); + + void addSecret(String name, String value); + + AssetReference addAsset(File file); + + AssetReference addAsset(String assetName, InputStream contents); + + void startConnector(); + + void stopConnector(); + + void waitForDataIngested(Duration maxWaitTime); + + void waitForIdle(Duration maxWaitTime); + + void waitForIdle(Duration minimumIdleTime, Duration maxWaitTime); + + List validate(); + + /** + * Returns the HTTP port on which the embedded Jetty server is listening, or -1 if no server is running. + * + * @return the HTTP port, or -1 if not applicable + */ + default int getHttpPort() { + return -1; + } + + List fetchAllowableValues(String stepName, String propertyName); + + /* + * Returns a {@link VersionedExternalFlow} representing the current state of the Active Flow Context. + * The Active Flow Context is the flow that is currently running (or most recently ran) in the Connector. + * This is useful for making assertions about how the flow is configured after updates have been applied. + * + * @return the VersionedExternalFlow for the Active Flow Context + */ + VersionedExternalFlow getActiveFlowSnapshot(); + + /** + * Returns a {@link VersionedExternalFlow} representing the current state of the Working Flow Context. + * The Working Flow Context is the flow that reflects configuration changes that have been made + * but not yet applied. This is useful for making assertions about how the flow will be configured + * once the update is applied. + * + * @return the VersionedExternalFlow for the Working Flow Context + */ + VersionedExternalFlow getWorkingFlowSnapshot(); +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server-nar/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-server-nar/pom.xml new file mode 100644 index 000000000000..4b970dc4867a --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server-nar/pom.xml @@ -0,0 +1,55 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-server-nar + nar + + + + org.apache.nifi + nifi-connector-mock-server + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-framework-nar + 2.9.0-SNAPSHOT + nar + + + + + + + + shibboleth + https://build.shibboleth.net/nexus/content/repositories/releases/ + + true + + + false + + + + \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-server/pom.xml new file mode 100644 index 000000000000..bcfb88f3e5fd --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/pom.xml @@ -0,0 +1,80 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-server + jar + + + + org.slf4j + slf4j-api + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-framework-core-api + 2.9.0-SNAPSHOT + provided + + + org.apache.nifi + nifi-framework-core + 2.9.0-SNAPSHOT + provided + + + org.apache.nifi + nifi-property-encryptor + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-connector-mock-api + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-server-api + 2.9.0-SNAPSHOT + + + org.eclipse.jetty + jetty-server + + + org.eclipse.jetty.ee11 + jetty-ee11-webapp + + + + + org.junit.jupiter + junit-jupiter + test + + + + \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockAuditService.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockAuditService.java new file mode 100644 index 000000000000..3aa7bbc543e3 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockAuditService.java @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.action.Action; +import org.apache.nifi.admin.service.AuditService; +import org.apache.nifi.history.History; +import org.apache.nifi.history.HistoryQuery; +import org.apache.nifi.history.PreviousValue; + +import java.util.Collection; +import java.util.Date; +import java.util.List; +import java.util.Map; + +public class MockAuditService implements AuditService { + @Override + public void addActions(final Collection actions) { + } + + @Override + public Map> getPreviousValues(final String componentId) { + return Map.of(); + } + + @Override + public void deletePreviousValues(final String propertyName, final String componentId) { + } + + @Override + public History getActions(final HistoryQuery actionQuery) { + return null; + } + + @Override + public History getActions(final int firstActionId, final int maxActions) { + return null; + } + + @Override + public Action getAction(final Integer actionId) { + return null; + } + + @Override + public void purgeActions(final Date end, final Action purgeAction) { + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManager.java new file mode 100644 index 000000000000..92892f6275a9 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManager.java @@ -0,0 +1,179 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.asset.AssetManagerInitializationContext; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Mock implementation of AssetManager for use in ConnectorTestRunner tests. + * This implementation stores assets in a temporary directory structure. + */ +public class MockConnectorAssetManager implements AssetManager { + + private static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory"; + private static final String DEFAULT_ASSET_STORAGE_LOCATION = "target/mock_connector_assets"; + + private final Map assets = new ConcurrentHashMap<>(); + private volatile File assetStorageLocation; + + @Override + public void initialize(final AssetManagerInitializationContext context) { + final String storageLocation = context.getProperties().getOrDefault(ASSET_STORAGE_LOCATION_PROPERTY, DEFAULT_ASSET_STORAGE_LOCATION); + assetStorageLocation = new File(storageLocation); + + if (!assetStorageLocation.exists()) { + try { + Files.createDirectories(assetStorageLocation.toPath()); + } catch (final IOException e) { + throw new RuntimeException("Failed to create asset storage directory: " + storageLocation, e); + } + } + } + + @Override + public Asset createAsset(final String ownerId, final String assetName, final InputStream contents) throws IOException { + final String assetId = UUID.randomUUID().toString(); + return saveAsset(ownerId, assetId, assetName, contents); + } + + @Override + public Asset saveAsset(final String ownerId, final String assetId, final String assetName, final InputStream contents) throws IOException { + final File assetFile = getFile(ownerId, assetId, assetName); + final File parentDir = assetFile.getParentFile(); + + if (!parentDir.exists()) { + Files.createDirectories(parentDir.toPath()); + } + + Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING); + + final Asset asset = new MockAsset(assetId, ownerId, assetName, assetFile, null); + assets.put(assetId, asset); + return asset; + } + + @Override + public Optional getAsset(final String id) { + return Optional.ofNullable(assets.get(id)); + } + + @Override + public List getAssets(final String ownerId) { + final List ownerAssets = new ArrayList<>(); + for (final Asset asset : assets.values()) { + if (asset.getOwnerIdentifier().equals(ownerId)) { + ownerAssets.add(asset); + } + } + return ownerAssets; + } + + @Override + public Asset createMissingAsset(final String ownerId, final String assetName) { + final String assetId = UUID.randomUUID().toString(); + final File file = getFile(ownerId, assetId, assetName); + final Asset asset = new MockAsset(assetId, ownerId, assetName, file, null); + assets.put(assetId, asset); + return asset; + } + + @Override + public Optional deleteAsset(final String id) { + final Asset removed = assets.remove(id); + if (removed != null && removed.getFile().exists()) { + try { + Files.delete(removed.getFile().toPath()); + } catch (final IOException e) { + throw new RuntimeException("Failed to delete asset " + id + " from storage file " + removed.getFile().getAbsolutePath(), e); + } + } + + return Optional.ofNullable(removed); + } + + private File getFile(final String ownerId, final String assetId, final String assetName) { + final Path parentPath = assetStorageLocation.toPath().normalize(); + final Path assetPath = Path.of(ownerId, assetId, assetName).normalize(); + return parentPath.resolve(assetPath).toFile(); + } + + /** + * A simple Asset implementation for the mock. + */ + private static class MockAsset implements Asset { + private final String identifier; + private final String ownerIdentifier; + private final String name; + private final File file; + private final String digest; + + MockAsset(final String identifier, final String ownerIdentifier, final String name, final File file, final String digest) { + this.identifier = identifier; + this.ownerIdentifier = ownerIdentifier; + this.name = name; + this.file = file; + this.digest = digest; + } + + @Override + public String getIdentifier() { + return identifier; + } + + @Override + @Deprecated + public String getParameterContextIdentifier() { + return ownerIdentifier; + } + + @Override + public String getOwnerIdentifier() { + return ownerIdentifier; + } + + @Override + public String getName() { + return name; + } + + @Override + public File getFile() { + return file; + } + + @Override + public Optional getDigest() { + return Optional.ofNullable(digest); + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorInitializationContext.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorInitializationContext.java new file mode 100644 index 000000000000..02d276bc28dd --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorInitializationContext.java @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.connector.BundleCompatibility; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.StandardConnectorInitializationContext; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +public class MockConnectorInitializationContext extends StandardConnectorInitializationContext { + + private final MockExtensionMapper mockExtensionMapper; + + protected MockConnectorInitializationContext(final Builder builder) { + super(builder); + this.mockExtensionMapper = builder.mockExtensionMapper; + } + + @Override + public void updateFlow(final FlowContext flowContext, final VersionedExternalFlow versionedExternalFlow, + final BundleCompatibility bundleCompatability) throws FlowUpdateException { + replaceMocks(versionedExternalFlow.getFlowContents()); + super.updateFlow(flowContext, versionedExternalFlow, bundleCompatability); + } + + private void replaceMocks(final VersionedProcessGroup group) { + if (group.getProcessors() != null) { + for (final VersionedProcessor processor : group.getProcessors()) { + mockExtensionMapper.mapProcessor(processor); + } + } + + if (group.getControllerServices() != null) { + for (final VersionedControllerService controllerService : group.getControllerServices()) { + mockExtensionMapper.mapControllerService(controllerService); + } + } + + if (group.getProcessGroups() != null) { + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + replaceMocks(childGroup); + } + } + } + + + public static class Builder extends StandardConnectorInitializationContext.Builder { + private final MockExtensionMapper mockExtensionMapper; + + public Builder(final MockExtensionMapper mockExtensionMapper) { + this.mockExtensionMapper = mockExtensionMapper; + } + + @Override + public MockConnectorInitializationContext build() { + return new MockConnectorInitializationContext(this); + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorRepository.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorRepository.java new file mode 100644 index 000000000000..9f951116747d --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockConnectorRepository.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.connector.FrameworkConnectorInitializationContextBuilder; +import org.apache.nifi.components.connector.StandardConnectorRepository; + +public class MockConnectorRepository extends StandardConnectorRepository { + + private volatile MockExtensionMapper mockExtensionMapper; + + public void setMockExtensionMapper(final MockExtensionMapper mapper) { + mockExtensionMapper = mapper; + } + + @Override + public FrameworkConnectorInitializationContextBuilder createInitializationContextBuilder() { + return new MockConnectorInitializationContext.Builder(mockExtensionMapper); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionDiscoveringManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionDiscoveringManager.java new file mode 100644 index 000000000000..22b2ca043ce6 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionDiscoveringManager.java @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.bundle.BundleDetails; +import org.apache.nifi.controller.ControllerService; +import org.apache.nifi.nar.InstanceClassLoader; +import org.apache.nifi.nar.StandardExtensionDiscoveringManager; +import org.apache.nifi.processor.Processor; + +import java.io.File; +import java.net.URL; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; + +public class MockExtensionDiscoveringManager extends StandardExtensionDiscoveringManager { + private final ConcurrentMap mockComponentClassLoaders = new ConcurrentHashMap<>(); + + public synchronized void addProcessor(final Class mockProcessorClass) { + final BundleDetails bundleDetails = new BundleDetails.Builder() + .workingDir(new File("target/work/extensions/mock-bundle")) + .coordinate(new BundleCoordinate("org.apache.nifi.mock", mockProcessorClass.getName(), "1.0.0")) + .build(); + + final Bundle mockBundle = new Bundle(bundleDetails, mockProcessorClass.getClassLoader()); + discoverExtensions(Set.of(mockBundle)); + + mockComponentClassLoaders.put(mockProcessorClass.getName(), mockProcessorClass.getClassLoader()); + registerExtensionClass(Processor.class, mockProcessorClass.getName(), mockBundle); + } + + public synchronized void addControllerService(final Class mockControllerServiceClass) { + final BundleDetails bundleDetails = new BundleDetails.Builder() + .workingDir(new File("target/work/extensions/mock-bundle")) + .coordinate(new BundleCoordinate("org.apache.nifi.mock", mockControllerServiceClass.getName(), "1.0.0")) + .build(); + + final Bundle mockBundle = new Bundle(bundleDetails, mockControllerServiceClass.getClassLoader()); + discoverExtensions(Set.of(mockBundle)); + + mockComponentClassLoaders.put(mockControllerServiceClass.getName(), mockControllerServiceClass.getClassLoader()); + registerExtensionClass(ControllerService.class, mockControllerServiceClass.getName(), mockBundle); + } + + @Override + public InstanceClassLoader createInstanceClassLoader(final String classType, final String instanceIdentifier, final Bundle bundle, final Set additionalUrls, + final boolean register, final String classloaderIsolationKey) { + + final ClassLoader classLoader = mockComponentClassLoaders.get(classType); + if (classLoader != null) { + return new InstanceClassLoader(instanceIdentifier, classType, additionalUrls, Set.of(), classLoader); + } + + return super.createInstanceClassLoader(classType, instanceIdentifier, bundle, additionalUrls, register, classloaderIsolationKey); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionMapper.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionMapper.java new file mode 100644 index 000000000000..edf81cdeaac5 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockExtensionMapper.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.HashMap; +import java.util.Map; + +public class MockExtensionMapper { + + private final Map processorMocks = new HashMap<>(); + private final Map controllerServiceMocks = new HashMap<>(); + + public void mockProcessor(final String processorType, final String mockProcessorClassName) { + processorMocks.put(processorType, mockProcessorClassName); + } + + public void mockControllerService(final String controllerServiceType, final String mockControllerServiceClassName) { + controllerServiceMocks.put(controllerServiceType, mockControllerServiceClassName); + } + + public void mapProcessor(final VersionedProcessor processor) { + final String type = processor.getType(); + final String implementationClassName = processorMocks.get(type); + if (implementationClassName == null) { + return; + } + + processor.setType(implementationClassName); + processor.setBundle(new Bundle("org.apache.nifi.mock", implementationClassName, "1.0.0")); + } + + public void mapControllerService(final VersionedControllerService controllerService) { + final String type = controllerService.getType(); + final String implementationClassName = controllerServiceMocks.get(type); + if (implementationClassName == null) { + return; + } + + controllerService.setType(implementationClassName); + controllerService.setBundle(new Bundle("org.apache.nifi.mock", implementationClassName, "1.0.0")); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContext.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContext.java new file mode 100644 index 000000000000..9e8891ee5270 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContext.java @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.web.NiFiConnectorWebContext; + +/** + * Mock implementation of {@link NiFiConnectorWebContext} for the connector test runner. + * Provides direct access to the connector instance and its flow contexts without + * authorization proxying, since the mock server uses a permit-all authorizer. + */ +public class MockNiFiConnectorWebContext implements NiFiConnectorWebContext { + + private final ConnectorRepository connectorRepository; + + public MockNiFiConnectorWebContext(final ConnectorRepository connectorRepository) { + this.connectorRepository = connectorRepository; + } + + @Override + @SuppressWarnings("unchecked") + public ConnectorWebContext getConnectorWebContext(final String connectorId) throws IllegalArgumentException { + final ConnectorNode connectorNode = connectorRepository.getConnector(connectorId); + if (connectorNode == null) { + throw new IllegalArgumentException("Unable to find connector with id: " + connectorId); + } + + final T connector = (T) connectorNode.getConnector(); + final FlowContext workingFlowContext = connectorNode.getWorkingFlowContext(); + final FlowContext activeFlowContext = connectorNode.getActiveFlowContext(); + + return new ConnectorWebContext<>(connector, workingFlowContext, activeFlowContext); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockRuleViolationsManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockRuleViolationsManager.java new file mode 100644 index 000000000000..bb0e2636ea5a --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockRuleViolationsManager.java @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.flow.VersionedComponent; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.validation.RuleViolation; +import org.apache.nifi.validation.RuleViolationsManager; + +import java.util.Collection; +import java.util.List; +import java.util.Map; + +public class MockRuleViolationsManager implements RuleViolationsManager { + @Override + public void upsertComponentViolations(final String subjectId, final Collection violations) { + + } + + @Override + public void upsertGroupViolations(final VersionedProcessGroup processGroup, final Collection violations, + final Map> componentToRuleViolations) { + } + + @Override + public Collection getRuleViolationsForSubject(final String subjectId) { + return List.of(); + } + + @Override + public Collection getRuleViolationsForGroup(final String groupId) { + return List.of(); + } + + @Override + public Collection getRuleViolationsForGroups(final Collection groupIds) { + return List.of(); + } + + @Override + public Collection getAllRuleViolations() { + return List.of(); + } + + @Override + public void removeRuleViolationsForSubject(final String subjectId) { + + } + + @Override + public void removeRuleViolationsForRule(final String ruleId) { + + } + + @Override + public void cleanUp() { + + } + + @Override + public boolean isEmpty() { + return false; + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockServerConfigVerificationResult.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockServerConfigVerificationResult.java new file mode 100644 index 000000000000..fed82f929020 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockServerConfigVerificationResult.java @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; + +import java.util.Collections; +import java.util.List; + +public class MockServerConfigVerificationResult implements ConnectorConfigVerificationResult { + private final List results; + + public MockServerConfigVerificationResult(final List results) { + this.results = results; + } + + @Override + public List getAllResults() { + return Collections.unmodifiableList(results); + } + + @Override + public List getFailedResults() { + return results.stream() + .filter(result -> result.getOutcome() == Outcome.FAILED) + .toList(); + } + + @Override + public void assertNoFailures() { + final List failedResults = getFailedResults(); + if (!failedResults.isEmpty()) { + throw new AssertionError("Configuration verification failed: " + failedResults); + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManager.java new file mode 100644 index 000000000000..f22269ceabfa --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManager.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; + +import java.io.IOException; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.atomic.AtomicLong; + +public class MockStateManager implements StateManager { + private final AtomicLong versionCounter = new AtomicLong(0L); + private final boolean dropStateKeySupported; + + private volatile StateMap localStateMap = new MockStateMap(null, -1L); + private volatile StateMap clusterStateMap = new MockStateMap(null, -1L); + + public MockStateManager(final boolean dropStateKeySupported) { + this.dropStateKeySupported = dropStateKeySupported; + } + + @Override + public synchronized void setState(final Map state, final Scope scope) throws IOException { + final StateMap newStateMap = new MockStateMap(state, versionCounter.incrementAndGet()); + + if (scope == Scope.CLUSTER) { + clusterStateMap = newStateMap; + } else { + localStateMap = newStateMap; + } + } + + @Override + public synchronized StateMap getState(final Scope scope) throws IOException { + if (scope == Scope.CLUSTER) { + return clusterStateMap; + } else { + return localStateMap; + } + } + + @Override + public synchronized boolean replace(final StateMap oldValue, final Map newValue, final Scope scope) throws IOException { + final StateMap currentStateMap = scope == Scope.CLUSTER ? clusterStateMap : localStateMap; + + if (currentStateMap == oldValue) { + final StateMap newStateMap = new MockStateMap(newValue, versionCounter.incrementAndGet()); + + if (scope == Scope.CLUSTER) { + clusterStateMap = newStateMap; + } else { + localStateMap = newStateMap; + } + + return true; + } + + return false; + } + + @Override + public synchronized void clear(final Scope scope) throws IOException { + setState(Collections.emptyMap(), scope); + } + + @Override + public boolean isStateKeyDropSupported() { + return dropStateKeySupported; + } +} + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManagerProvider.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManagerProvider.java new file mode 100644 index 000000000000..77484904f251 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateManagerProvider.java @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateManagerProvider; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; + +public class MockStateManagerProvider implements StateManagerProvider { + private final ConcurrentMap stateManagers = new ConcurrentHashMap<>(); + private volatile boolean clusterProviderEnabled = false; + + @Override + public StateManager getStateManager(final String componentId, final boolean dropStateKeySupported) { + return stateManagers.computeIfAbsent(componentId, id -> new MockStateManager(dropStateKeySupported)); + } + + @Override + public void onComponentRemoved(final String componentId) { + stateManagers.remove(componentId); + } + + @Override + public void shutdown() { + stateManagers.clear(); + } + + @Override + public void enableClusterProvider() { + clusterProviderEnabled = true; + } + + @Override + public void disableClusterProvider() { + clusterProviderEnabled = false; + } + + @Override + public boolean isClusterProviderEnabled() { + return clusterProviderEnabled; + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateMap.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateMap.java new file mode 100644 index 000000000000..54b0980d6162 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/MockStateMap.java @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.state.StateMap; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +public class MockStateMap implements StateMap { + private final Map stateValues; + private final long version; + + public MockStateMap(final Map stateValues, final long version) { + this.stateValues = stateValues == null ? Collections.emptyMap() : new HashMap<>(stateValues); + this.version = version; + } + + @Override + public Optional getStateVersion() { + return version == -1L ? Optional.empty() : Optional.of(Long.toString(version)); + } + + @Override + public String get(final String key) { + return stateValues.get(key); + } + + @Override + public Map toMap() { + return Collections.unmodifiableMap(stateValues); + } + + @Override + public String toString() { + return "MockStateMap[version=" + version + ", values=" + stateValues + "]"; + } +} + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/NopPropertyEncryptor.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/NopPropertyEncryptor.java new file mode 100644 index 000000000000..56fbe816b7f3 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/NopPropertyEncryptor.java @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.encrypt.PropertyEncryptor; + +public class NopPropertyEncryptor implements PropertyEncryptor { + @Override + public String encrypt(final String property) { + return property; + } + + @Override + public String decrypt(final String encryptedProperty) { + return encryptedProperty; + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/PermitAllAuthorizer.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/PermitAllAuthorizer.java new file mode 100644 index 000000000000..75b2577f25e5 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/PermitAllAuthorizer.java @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.authorization.AuthorizationRequest; +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.AuthorizerConfigurationContext; +import org.apache.nifi.authorization.AuthorizerInitializationContext; +import org.apache.nifi.authorization.exception.AuthorizationAccessException; +import org.apache.nifi.authorization.exception.AuthorizerCreationException; +import org.apache.nifi.authorization.exception.AuthorizerDestructionException; + +public class PermitAllAuthorizer implements Authorizer { + @Override + public AuthorizationResult authorize(final AuthorizationRequest request) throws AuthorizationAccessException { + return AuthorizationResult.approved(); + } + + @Override + public void initialize(final AuthorizerInitializationContext initializationContext) throws AuthorizerCreationException { + } + + @Override + public void onConfigured(final AuthorizerConfigurationContext configurationContext) throws AuthorizerCreationException { + } + + @Override + public void preDestruction() throws AuthorizerDestructionException { + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServer.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServer.java new file mode 100644 index 000000000000..d491d0e6c19e --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServer.java @@ -0,0 +1,629 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import jakarta.servlet.ServletContext; +import org.apache.nifi.admin.service.AuditService; +import org.apache.nifi.asset.Asset; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.bundle.BundleDetails; +import org.apache.nifi.cluster.ClusterDetailsFactory; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.GhostConnector; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StandaloneConnectorRequestReplicator; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.components.connector.StringLiteralValue; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.components.state.StateManagerProvider; +import org.apache.nifi.components.validation.DisabledServiceValidationResult; +import org.apache.nifi.components.validation.ValidationState; +import org.apache.nifi.connectable.FlowFileTransferCounts; +import org.apache.nifi.controller.ControllerService; +import org.apache.nifi.controller.DecommissionTask; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.metrics.DefaultComponentMetricReporter; +import org.apache.nifi.controller.repository.FlowFileEventRepository; +import org.apache.nifi.controller.repository.metrics.RingBufferEventRepository; +import org.apache.nifi.controller.status.history.StatusHistoryDumpFactory; +import org.apache.nifi.controller.status.history.StatusHistoryRepository; +import org.apache.nifi.controller.status.history.VolatileComponentStatusRepository; +import org.apache.nifi.diagnostics.DiagnosticsFactory; +import org.apache.nifi.encrypt.PropertyEncryptor; +import org.apache.nifi.engine.FlowEngine; +import org.apache.nifi.events.VolatileBulletinRepository; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.mock.connector.server.secrets.ConnectorTestRunnerSecretsManager; +import org.apache.nifi.nar.ExtensionMapping; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.processor.Processor; +import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; +import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.validation.RuleViolationsManager; +import org.apache.nifi.web.NiFiConnectorWebContext; +import org.eclipse.jetty.ee.webapp.WebAppClassLoader; +import org.eclipse.jetty.ee11.webapp.WebAppContext; +import org.eclipse.jetty.server.Server; +import org.eclipse.jetty.server.ServerConnector; +import org.eclipse.jetty.server.handler.ContextHandlerCollection; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.jar.JarFile; +import java.util.stream.Stream; + +public class StandardConnectorMockServer implements ConnectorMockServer { + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorMockServer.class); + + private static final String CONNECTOR_ID = "test-connector"; + private static final String NAR_DEPENDENCIES_PATH = "NAR-INF/bundled-dependencies"; + private static final String CONNECTOR_WAR_MANIFEST_PATH = "META-INF/nifi-connector"; + private static final String WAR_EXTENSION = ".war"; + + private Bundle systemBundle; + private Set bundles; + private NiFiProperties nifiProperties; + private FlowController flowController; + private MockExtensionDiscoveringManager extensionManager; + private ConnectorNode connectorNode; + private ConnectorRepository connectorRepository; + private FlowEngine flowEngine; + private MockExtensionMapper mockExtensionMapper; + private FlowFileTransferCounts initialFlowFileTransferCounts = new FlowFileTransferCounts(0L, 0L, 0L, 0L); + private Server jettyServer; + + @Override + public void start() { + extensionManager = new MockExtensionDiscoveringManager(); + extensionManager.discoverExtensions(systemBundle, bundles); + extensionManager.logClassLoaderMapping(); + + final FlowFileEventRepository flowFileEventRepository = new RingBufferEventRepository(5); + final Authorizer authorizer = new PermitAllAuthorizer(); + final AuditService auditService = new MockAuditService(); + final PropertyEncryptor propertyEncryptor = new NopPropertyEncryptor(); + final BulletinRepository bulletinRepository = new VolatileBulletinRepository(); + final StatusHistoryRepository statusHistoryRepository = new VolatileComponentStatusRepository(nifiProperties); + final RuleViolationsManager ruleViolationManager = new MockRuleViolationsManager(); + final StateManagerProvider stateManagerProvider = new MockStateManagerProvider(); + + flowController = FlowController.createStandaloneInstance( + flowFileEventRepository, + null, + nifiProperties, + authorizer, + auditService, + new DefaultComponentMetricReporter(), + propertyEncryptor, + bulletinRepository, + extensionManager, + statusHistoryRepository, + ruleViolationManager, + stateManagerProvider, + new StandaloneConnectorRequestReplicator()); + + try { + flowController.getRepositoryContextFactory().getFlowFileRepository().loadFlowFiles(Collections::emptyList); + } catch (final IOException e) { + throw new RuntimeException("Failed to initialize FlowFile Repository", e); + } + + connectorRepository = flowController.getConnectorRepository(); + if (!(connectorRepository instanceof MockConnectorRepository)) { + throw new IllegalStateException("Connector Repository is not an instance of MockConnectorRepository"); + } + + mockExtensionMapper = new MockExtensionMapper(); + ((MockConnectorRepository) connectorRepository).setMockExtensionMapper(mockExtensionMapper); + + flowEngine = new FlowEngine(4, "Connector Threads"); + + startJettyServer(); + } + + @Override + public void initialize(final NiFiProperties properties, final Bundle systemBundle, final Set bundles, final ExtensionMapping extensionMapping) { + this.systemBundle = systemBundle; + this.bundles = bundles; + this.nifiProperties = properties; + } + + @Override + public void stop() { + if (jettyServer != null) { + try { + jettyServer.stop(); + logger.info("Jetty server stopped"); + } catch (final Exception e) { + logger.warn("Failed to stop Jetty server", e); + } + } + if (flowEngine != null) { + flowEngine.shutdown(); + } + if (flowController != null) { + flowController.shutdown(false); + } + } + + @Override + public DiagnosticsFactory getDiagnosticsFactory() { + return null; + } + + @Override + public DiagnosticsFactory getThreadDumpFactory() { + return null; + } + + @Override + public DecommissionTask getDecommissionTask() { + return null; + } + + @Override + public ClusterDetailsFactory getClusterDetailsFactory() { + return null; + } + + @Override + public StatusHistoryDumpFactory getStatusHistoryDumpFactory() { + return null; + } + + @Override + public void instantiateConnector(final String connectorClassName) { + final List bundles = extensionManager.getBundles(connectorClassName); + if (bundles.isEmpty()) { + throw new IllegalStateException("No bundles found for connector class: " + connectorClassName + " - ensure that you have included all relevant NARs in the configured lib directory"); + } + if (bundles.size() > 1) { + throw new IllegalStateException("Multiple bundles found for connector class: " + connectorClassName + " - unable to determine which bundle to use. Ensure that only a single version of " + + "the Connector is included in the configured lib directory. Available bundles: " + bundles); + } + + final BundleCoordinate bundleCoordinate = bundles.getFirst().getBundleDetails().getCoordinate(); + connectorNode = flowController.getFlowManager().createConnector(connectorClassName, CONNECTOR_ID, bundleCoordinate, true, true); + + if (connectorNode.isExtensionMissing()) { + final Connector connector = connectorNode.getConnector(); + if (connector instanceof final GhostConnector ghostConnector) { + throw new IllegalStateException("Failed to create Connector of type " + connectorClassName, ghostConnector.getCauseOfGhost()); + } else { + throw new IllegalStateException("Failed to create Connector of type " + connectorClassName); + } + } + } + + @Override + public void registerMockBundle(final ClassLoader classLoader, final File workingDirectory) { + new BundleDetails.Builder() + .workingDir(workingDirectory) + .coordinate(new BundleCoordinate("org.apache.nifi.mock", "nifi-connector-mock-bundle", "1.0.0")) + .build(); + } + + @Override + public void applyUpdate() throws FlowUpdateException { + final ConnectorState initialDesiredState = connectorNode.getDesiredState(); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.applyUpdate(); + + if (initialDesiredState == ConnectorState.RUNNING) { + connectorNode.start(flowEngine); + } else { + connectorNode.stop(flowEngine); + } + } + + @Override + public void configure(final String stepName, final StepConfiguration configuration) throws FlowUpdateException { + connectorNode.setConfiguration(stepName, configuration); + } + + @Override + public void configure(final String stepName, final Map propertyValues) throws FlowUpdateException { + configure(stepName, propertyValues, Collections.emptyMap()); + } + + @Override + public void configure(final String stepName, final Map propertyValues, final Map propertyReferences) throws FlowUpdateException { + final StepConfiguration stepConfiguration = createStepConfiguration(propertyValues, propertyReferences); + configure(stepName, stepConfiguration); + } + + @Override + public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map propertyValueOverrides) { + return verifyConfiguration(stepName, propertyValueOverrides, Collections.emptyMap()); + } + + @Override + public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map propertyValueOverrides, + final Map referenceOverrides) { + + final StepConfiguration configuration = createStepConfiguration(propertyValueOverrides, referenceOverrides); + return verifyConfiguration(stepName, configuration); + } + + private StepConfiguration createStepConfiguration(final Map propertyValues, final Map propertyReferences) { + final Map references = new HashMap<>(); + propertyValues.forEach((key, value) -> references.put(key, value == null ? null : new StringLiteralValue(value))); + references.putAll(propertyReferences); + return new StepConfiguration(references); + } + + @Override + public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final StepConfiguration configurationOverrides) { + final List results = connectorNode.verifyConfigurationStep(stepName, configurationOverrides); + return new MockServerConfigVerificationResult(results); + } + + @Override + public void addSecret(final String name, final String value) { + final SecretsManager secretsManager = connectorRepository.getSecretsManager(); + if (!(secretsManager instanceof final ConnectorTestRunnerSecretsManager testRunnerSecretsManager)) { + throw new IllegalStateException("Secrets Manager is not an instance of ConnectorTestRunnerSecretsManager"); + } + + testRunnerSecretsManager.addSecret(name, value); + } + + @Override + public SecretReference createSecretReference(final String secretName) { + return new SecretReference(ConnectorTestRunner.SECRET_PROVIDER_ID, ConnectorTestRunner.SECRET_PROVIDER_NAME, secretName, secretName); + } + + @Override + public AssetReference addAsset(final File file) { + final AssetManager assetManager = flowController.getConnectorAssetManager(); + + try (final InputStream inputStream = new FileInputStream(file)) { + final Asset asset = assetManager.createAsset(CONNECTOR_ID, file.getName(), inputStream); + return new AssetReference(Set.of(asset.getIdentifier())); + } catch (final IOException e) { + throw new RuntimeException("Failed to add asset from file: " + file.getAbsolutePath(), e); + } + } + + @Override + public AssetReference addAsset(final String assetName, final InputStream contents) { + final AssetManager assetManager = flowController.getConnectorAssetManager(); + + try { + final Asset asset = assetManager.createAsset(CONNECTOR_ID, assetName, contents); + return new AssetReference(Set.of(asset.getIdentifier())); + } catch (final IOException e) { + throw new RuntimeException("Failed to add asset: " + assetName, e); + } + } + + @Override + public void startConnector() { + initialFlowFileTransferCounts = connectorNode.getFlowFileTransferCounts(); + + connectorNode.start(flowEngine); + } + + @Override + public void stopConnector() { + try { + connectorNode.stop(flowEngine).get(10, TimeUnit.SECONDS); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting for connector to stop", e); + } catch (final Exception e) { + throw new RuntimeException("Failed to stop Connector", e); + } + } + + @Override + public void waitForDataIngested(final Duration maxWaitTime) { + final long startTime = System.currentTimeMillis(); + final long expirationTime = startTime + maxWaitTime.toMillis(); + + while (getTransferCountsSinceStart().getReceivedCount() == 0L) { + if (System.currentTimeMillis() > expirationTime) { + throw new RuntimeException("Timed out waiting for data to be ingested by the Connector"); + } + + try { + Thread.sleep(100L); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting for data to be ingested by the Connector", e); + } + } + } + + private FlowFileTransferCounts getTransferCountsSinceStart() { + final FlowFileTransferCounts currentCounts = connectorNode.getFlowFileTransferCounts(); + return currentCounts.minus(initialFlowFileTransferCounts); + } + + @Override + public void waitForIdle(final Duration maxWaitTime) { + waitForIdle(Duration.ofMillis(0L), maxWaitTime); + } + + @Override + public void waitForIdle(final Duration minimumIdleTime, final Duration maxWaitTime) { + Optional idleTime = connectorNode.getIdleDuration(); + + // Wait until idleTime is not empty and is at least equal to minimumIdleTime + final long startTime = System.currentTimeMillis(); + final long expirationTime = startTime + maxWaitTime.toMillis(); + + while (idleTime.isEmpty() || idleTime.get().compareTo(minimumIdleTime) <= 0) { + if (System.currentTimeMillis() > expirationTime) { + throw new RuntimeException("Timed out waiting for Connector to be idle"); + } + + try { + Thread.sleep(100L); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting for Connector to be idle", e); + } + + idleTime = connectorNode.getIdleDuration(); + } + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName) { + return connectorNode.fetchAllowableValues(stepName, propertyName); + } + + @Override + public List validate() { + final ValidationState validationState = connectorNode.performValidation(); + return validationState.getValidationErrors().stream() + .filter(result -> !result.isValid()) + .filter(result -> !DisabledServiceValidationResult.isMatch(result)) + .toList(); + } + + @Override + public VersionedExternalFlow getActiveFlowSnapshot() { + final FrameworkFlowContext activeFlowContext = connectorNode.getActiveFlowContext(); + if (activeFlowContext == null) { + throw new IllegalStateException("Active Flow Context is not available. The Connector may not have been initialized or may not have an initial flow."); + } + return createFlowSnapshot(activeFlowContext); + } + + @Override + public VersionedExternalFlow getWorkingFlowSnapshot() { + final FrameworkFlowContext workingFlowContext = connectorNode.getWorkingFlowContext(); + if (workingFlowContext == null) { + throw new IllegalStateException("Working Flow Context is not available. No configuration changes may have been made since the last update was applied."); + } + return createFlowSnapshot(workingFlowContext); + } + + private VersionedExternalFlow createFlowSnapshot(final FrameworkFlowContext flowContext) { + final ProcessGroup processGroup = flowContext.getManagedProcessGroup(); + + final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder() + .mapSensitiveConfiguration(false) + .mapPropertyDescriptors(true) + .stateLookup(VersionedComponentStateLookup.ENABLED_OR_DISABLED) + .sensitiveValueEncryptor(value -> value) + .componentIdLookup(ComponentIdLookup.VERSIONED_OR_GENERATE) + .mapInstanceIdentifiers(true) + .mapControllerServiceReferencesToVersionedId(true) + .mapFlowRegistryClientId(true) + .mapAssetReferences(true) + .build(); + + final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(extensionManager, flowMappingOptions); + final VersionedProcessGroup versionedGroup = flowMapper.mapProcessGroup( + processGroup, flowController.getControllerServiceProvider(), flowController.getFlowManager(), true); + + final VersionedExternalFlow externalFlow = new VersionedExternalFlow(); + externalFlow.setFlowContents(versionedGroup); + + final ParameterContext parameterContext = processGroup.getParameterContext(); + if (parameterContext != null) { + final Map parameterContexts = new HashMap<>(); + final VersionedParameterContext versionedParameterContext = createVersionedParameterContext(parameterContext); + parameterContexts.put(versionedParameterContext.getName(), versionedParameterContext); + externalFlow.setParameterContexts(parameterContexts); + } + + return externalFlow; + } + + private VersionedParameterContext createVersionedParameterContext(final ParameterContext parameterContext) { + final VersionedParameterContext versionedParameterContext = new VersionedParameterContext(); + versionedParameterContext.setName(parameterContext.getName()); + versionedParameterContext.setDescription(parameterContext.getDescription()); + versionedParameterContext.setIdentifier(parameterContext.getIdentifier()); + + final Set versionedParameters = new LinkedHashSet<>(); + for (final Parameter parameter : parameterContext.getParameters().values()) { + final VersionedParameter versionedParameter = new VersionedParameter(); + versionedParameter.setName(parameter.getDescriptor().getName()); + versionedParameter.setDescription(parameter.getDescriptor().getDescription()); + versionedParameter.setSensitive(parameter.getDescriptor().isSensitive()); + versionedParameter.setProvided(parameter.getDescriptor().isSensitive()); + if (!parameter.getDescriptor().isSensitive()) { + versionedParameter.setValue(parameter.getValue()); + } + versionedParameters.add(versionedParameter); + } + versionedParameterContext.setParameters(versionedParameters); + + return versionedParameterContext; + } + + @Override + public void mockProcessor(final String processorType, final Class mockProcessorClass) { + mockExtensionMapper.mockProcessor(processorType, mockProcessorClass.getName()); + extensionManager.addProcessor(mockProcessorClass); + } + + @Override + public void mockControllerService(final String controllerServiceType, final Class mockControllerServiceClass) { + mockExtensionMapper.mockControllerService(controllerServiceType, mockControllerServiceClass.getName()); + extensionManager.addControllerService(mockControllerServiceClass); + } + + @Override + public int getHttpPort() { + if (jettyServer == null) { + return -1; + } + + final ServerConnector connector = (ServerConnector) jettyServer.getConnectors()[0]; + return connector.getLocalPort(); + } + + @Override + public void close() { + stop(); + } + + private void startJettyServer() { + final String httpPortValue = nifiProperties.getProperty(NiFiProperties.WEB_HTTP_PORT); + if (httpPortValue == null || httpPortValue.isBlank()) { + logger.debug("No HTTP port configured; skipping Jetty server startup"); + return; + } + + final int httpPort = Integer.parseInt(httpPortValue); + final Map wars = findWars(bundles); + if (wars.isEmpty()) { + logger.debug("No WAR files found in NAR bundles; skipping Jetty server startup"); + return; + } + + jettyServer = new Server(); + + final ServerConnector serverConnector = new ServerConnector(jettyServer); + serverConnector.setPort(httpPort); + jettyServer.addConnector(serverConnector); + + final List webAppContexts = new ArrayList<>(); + final ContextHandlerCollection handlers = new ContextHandlerCollection(); + for (final Map.Entry entry : wars.entrySet()) { + final File warFile = entry.getKey(); + final Bundle bundle = entry.getValue(); + + final String warName = warFile.getName(); + final String contextPath = "/" + warName.substring(0, warName.length() - WAR_EXTENSION.length()); + + final WebAppContext webAppContext = new WebAppContext(warFile.getPath(), contextPath); + webAppContext.setClassLoader(new WebAppClassLoader(bundle.getClassLoader(), webAppContext)); + + handlers.addHandler(webAppContext); + webAppContexts.add(webAppContext); + logger.info("Deploying WAR [{}] at context path [{}]", warFile.getAbsolutePath(), contextPath); + } + + jettyServer.setHandler(handlers); + + try { + jettyServer.start(); + logger.info("Jetty server started on port [{}]", getHttpPort()); + } catch (final Exception e) { + throw new RuntimeException("Failed to start Jetty server", e); + } + + performInjectionForConnectorUis(webAppContexts); + } + + private void performInjectionForConnectorUis(final List webAppContexts) { + final NiFiConnectorWebContext connectorWebContext = new MockNiFiConnectorWebContext(connectorRepository); + for (final WebAppContext webAppContext : webAppContexts) { + final ServletContext servletContext = webAppContext.getServletHandler().getServletContext(); + servletContext.setAttribute("nifi-connector-web-context", connectorWebContext); + logger.info("Injected NiFiConnectorWebContext into WAR context [{}]", webAppContext.getContextPath()); + } + } + + public Map findWars(final Set bundles) { + final Map wars = new HashMap<>(); + + bundles.forEach(bundle -> { + final BundleDetails details = bundle.getBundleDetails(); + final Path bundledDependencies = new File(details.getWorkingDirectory(), NAR_DEPENDENCIES_PATH).toPath(); + if (Files.isDirectory(bundledDependencies)) { + try (final Stream dependencies = Files.list(bundledDependencies)) { + dependencies.filter(dependency -> dependency.getFileName().toString().endsWith(WAR_EXTENSION)) + .map(Path::toFile) + .filter(this::isConnectorWar) + .forEach(dependency -> wars.put(dependency, bundle)); + } catch (final IOException e) { + logger.warn("Failed to find WAR files in bundled-dependencies [{}]", bundledDependencies, e); + } + } + }); + + return wars; + } + + private boolean isConnectorWar(final File warFile) { + try (final JarFile jarFile = new JarFile(warFile)) { + return jarFile.getJarEntry(CONNECTOR_WAR_MANIFEST_PATH) != null; + } catch (final IOException e) { + logger.warn("Unable to inspect WAR file [{}] for connector manifest", warFile, e); + return false; + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerAuthorizable.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerAuthorizable.java new file mode 100644 index 000000000000..5600c34f143e --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerAuthorizable.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server.secrets; + +import org.apache.nifi.authorization.Resource; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.resource.ResourceFactory; + +public class ConnectorTestRunnerAuthorizable implements Authorizable { + @Override + public Authorizable getParentAuthorizable() { + return null; + } + + @Override + public Resource getResource() { + return ResourceFactory.getControllerResource(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretProvider.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretProvider.java new file mode 100644 index 000000000000..e12a3711bc1e --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretProvider.java @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server.secrets; + +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.secrets.SecretProvider; +import org.apache.nifi.components.connector.secrets.StandardSecret; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +public class ConnectorTestRunnerSecretProvider implements SecretProvider { + public static final String GROUP_NAME = "Default"; + private static final Authorizable AUTHORIZABLE = new ConnectorTestRunnerAuthorizable(); + + private final Map secrets = new HashMap<>(); + + public void addSecret(final String key, final String value) { + this.secrets.put(key, value); + } + + @Override + public String getProviderId() { + return ConnectorTestRunner.SECRET_PROVIDER_ID; + } + + @Override + public String getProviderName() { + return ConnectorTestRunner.SECRET_PROVIDER_ID; + } + + @Override + public List getAllSecrets() { + final List secrets = new ArrayList<>(); + for (final Map.Entry entry : this.secrets.entrySet()) { + final Secret secret = new StandardSecret.Builder() + .providerId(ConnectorTestRunner.SECRET_PROVIDER_ID) + .providerName(ConnectorTestRunner.SECRET_PROVIDER_NAME) + .groupName(GROUP_NAME) + .name(entry.getKey()) + .value(entry.getValue()) + .authorizable(AUTHORIZABLE) + .fullyQualifiedName(GROUP_NAME + "." + entry.getKey()) + .build(); + + secrets.add(secret); + } + + return secrets; + } + + @Override + public List getSecrets(final List fullyQualifiedSecretNames) { + final List matchingSecrets = new ArrayList<>(); + + for (final String secretName : fullyQualifiedSecretNames) { + final String value = secrets.get(secretName); + + if (value != null) { + final Secret secret = new StandardSecret.Builder() + .providerId(ConnectorTestRunner.SECRET_PROVIDER_ID) + .providerName(ConnectorTestRunner.SECRET_PROVIDER_NAME) + .groupName(GROUP_NAME) + .name(secretName) + .fullyQualifiedName(GROUP_NAME + "." + secretName) + .value(value) + .authorizable(AUTHORIZABLE) + .build(); + + matchingSecrets.add(secret); + } + } + return matchingSecrets; + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretsManager.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretsManager.java new file mode 100644 index 000000000000..e06f228454af --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/java/org/apache/nifi/mock/connector/server/secrets/ConnectorTestRunnerSecretsManager.java @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server.secrets; + +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.secrets.SecretProvider; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.components.connector.secrets.SecretsManagerInitializationContext; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +public class ConnectorTestRunnerSecretsManager implements SecretsManager { + private final ConnectorTestRunnerSecretProvider secretProvider = new ConnectorTestRunnerSecretProvider(); + + @Override + public void initialize(final SecretsManagerInitializationContext initializationContext) { + } + + public void addSecret(final String name, final String value) { + secretProvider.addSecret(name, value); + } + + @Override + public List getAllSecrets() { + return secretProvider.getAllSecrets(); + } + + @Override + public Set getSecretProviders() { + return Set.of(secretProvider); + } + + @Override + public Optional getSecret(final SecretReference secretReference) { + // Check that appropriate provider given + final SecretProvider provider = getProvider(secretReference); + if (provider == null) { + return Optional.empty(); + } + + final List secrets = provider.getSecrets(List.of(secretReference.getFullyQualifiedName())); + return secrets.isEmpty() ? Optional.empty() : Optional.of(secrets.getFirst()); + } + + private SecretProvider getProvider(final SecretReference secretReference) { + final String providerId = secretReference.getProviderId(); + if (providerId != null) { + for (final SecretProvider provider : getSecretProviders()) { + if (provider.getProviderId().equals(providerId)) { + return provider; + } + } + + return null; + } + + final String providerName = secretReference.getProviderName(); + if (providerName != null) { + for (final SecretProvider provider : getSecretProviders()) { + if (provider.getProviderName().equals(providerName)) { + return provider; + } + } + } + + return null; + } + + @Override + public Map getSecrets(final Set secretReferences) { + final Map secrets = new HashMap<>(); + for (final SecretReference reference : secretReferences) { + final Secret secret = getSecret(reference).orElse(null); + secrets.put(reference, secret); + } + + return secrets; + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.NiFiServer b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.NiFiServer new file mode 100644 index 000000000000..70572a99eaec --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.NiFiServer @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.mock.connector.server.StandardConnectorMockServer diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager new file mode 100644 index 000000000000..c23d0b17077c --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.mock.connector.server.MockConnectorAssetManager diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository new file mode 100644 index 000000000000..d9230afaec58 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.mock.connector.server.MockConnectorRepository diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager new file mode 100644 index 000000000000..a1c0de54d1f1 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.mock.connector.server.secrets.ConnectorTestRunnerSecretsManager diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManagerTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManagerTest.java new file mode 100644 index 000000000000..29521787641b --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockConnectorAssetManagerTest.java @@ -0,0 +1,245 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.asset.AssetManagerInitializationContext; +import org.apache.nifi.asset.AssetReferenceLookup; +import org.apache.nifi.controller.NodeTypeProvider; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MockConnectorAssetManagerTest { + + private static final String CONNECTOR_ID = "test-connector"; + private static final String ASSET_NAME = "test-asset.txt"; + private static final String ASSET_CONTENTS = "test-contents"; + + @TempDir + private Path tempDir; + + private MockConnectorAssetManager assetManager; + + @BeforeEach + void setUp() { + assetManager = new MockConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(tempDir); + assetManager.initialize(context); + } + + @Test + void testCreateAsset() throws IOException { + final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8)); + + final Asset asset = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents); + + assertNotNull(asset); + assertNotNull(asset.getIdentifier()); + assertEquals(CONNECTOR_ID, asset.getOwnerIdentifier()); + assertEquals(ASSET_NAME, asset.getName()); + assertTrue(asset.getFile().exists()); + assertEquals(ASSET_CONTENTS, Files.readString(asset.getFile().toPath(), StandardCharsets.UTF_8)); + } + + @Test + void testCreateAssetGeneratesUniqueIdentifiers() throws IOException { + final InputStream contents1 = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8)); + final InputStream contents2 = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8)); + + final Asset asset1 = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents1); + final Asset asset2 = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents2); + + assertNotEquals(asset1.getIdentifier(), asset2.getIdentifier()); + } + + @Test + void testSaveAsset() throws IOException { + final String assetId = "specific-asset-id"; + final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8)); + + final Asset asset = assetManager.saveAsset(CONNECTOR_ID, assetId, ASSET_NAME, contents); + + assertNotNull(asset); + assertEquals(assetId, asset.getIdentifier()); + assertEquals(CONNECTOR_ID, asset.getOwnerIdentifier()); + assertEquals(ASSET_NAME, asset.getName()); + assertTrue(asset.getFile().exists()); + assertEquals(ASSET_CONTENTS, Files.readString(asset.getFile().toPath(), StandardCharsets.UTF_8)); + } + + @Test + void testSaveAssetOverwritesExisting() throws IOException { + final String assetId = "overwrite-asset-id"; + final String originalContents = "original"; + final String updatedContents = "updated"; + + final InputStream originalStream = new ByteArrayInputStream(originalContents.getBytes(StandardCharsets.UTF_8)); + final Asset originalAsset = assetManager.saveAsset(CONNECTOR_ID, assetId, ASSET_NAME, originalStream); + assertEquals(originalContents, Files.readString(originalAsset.getFile().toPath(), StandardCharsets.UTF_8)); + + final InputStream updatedStream = new ByteArrayInputStream(updatedContents.getBytes(StandardCharsets.UTF_8)); + final Asset updatedAsset = assetManager.saveAsset(CONNECTOR_ID, assetId, ASSET_NAME, updatedStream); + assertEquals(updatedContents, Files.readString(updatedAsset.getFile().toPath(), StandardCharsets.UTF_8)); + } + + @Test + void testGetAsset() throws IOException { + final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8)); + final Asset createdAsset = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents); + + final Optional retrieved = assetManager.getAsset(createdAsset.getIdentifier()); + + assertTrue(retrieved.isPresent()); + assertEquals(createdAsset.getIdentifier(), retrieved.get().getIdentifier()); + assertEquals(createdAsset.getName(), retrieved.get().getName()); + assertEquals(createdAsset.getOwnerIdentifier(), retrieved.get().getOwnerIdentifier()); + } + + @Test + void testGetAssetReturnsEmptyForNonExistent() { + final Optional retrieved = assetManager.getAsset("non-existent-id"); + + assertTrue(retrieved.isEmpty()); + } + + @Test + void testGetAssets() throws IOException { + final InputStream contents1 = new ByteArrayInputStream("content1".getBytes(StandardCharsets.UTF_8)); + final InputStream contents2 = new ByteArrayInputStream("content2".getBytes(StandardCharsets.UTF_8)); + final InputStream contents3 = new ByteArrayInputStream("content3".getBytes(StandardCharsets.UTF_8)); + + assetManager.createAsset(CONNECTOR_ID, "asset1.txt", contents1); + assetManager.createAsset(CONNECTOR_ID, "asset2.txt", contents2); + assetManager.createAsset("other-connector", "asset3.txt", contents3); + + final List connectorAssets = assetManager.getAssets(CONNECTOR_ID); + assertEquals(2, connectorAssets.size()); + assertTrue(connectorAssets.stream().allMatch(a -> a.getOwnerIdentifier().equals(CONNECTOR_ID))); + + final List otherAssets = assetManager.getAssets("other-connector"); + assertEquals(1, otherAssets.size()); + assertEquals("other-connector", otherAssets.get(0).getOwnerIdentifier()); + } + + @Test + void testGetAssetsReturnsEmptyListForUnknownOwner() { + final List assets = assetManager.getAssets("unknown-owner"); + + assertNotNull(assets); + assertTrue(assets.isEmpty()); + } + + @Test + void testCreateMissingAsset() { + final Asset missingAsset = assetManager.createMissingAsset(CONNECTOR_ID, ASSET_NAME); + + assertNotNull(missingAsset); + assertNotNull(missingAsset.getIdentifier()); + assertEquals(CONNECTOR_ID, missingAsset.getOwnerIdentifier()); + assertEquals(ASSET_NAME, missingAsset.getName()); + assertFalse(missingAsset.getFile().exists()); + assertTrue(missingAsset.getDigest().isEmpty()); + } + + @Test + void testCreateMissingAssetCanBeRetrieved() { + final Asset missingAsset = assetManager.createMissingAsset(CONNECTOR_ID, ASSET_NAME); + + final Optional retrieved = assetManager.getAsset(missingAsset.getIdentifier()); + assertTrue(retrieved.isPresent()); + assertEquals(missingAsset.getIdentifier(), retrieved.get().getIdentifier()); + + final List ownerAssets = assetManager.getAssets(CONNECTOR_ID); + assertEquals(1, ownerAssets.size()); + } + + @Test + void testDeleteAsset() throws IOException { + final InputStream contents = new ByteArrayInputStream(ASSET_CONTENTS.getBytes(StandardCharsets.UTF_8)); + final Asset createdAsset = assetManager.createAsset(CONNECTOR_ID, ASSET_NAME, contents); + final Path assetFilePath = createdAsset.getFile().toPath(); + assertTrue(Files.exists(assetFilePath)); + + final Optional deleted = assetManager.deleteAsset(createdAsset.getIdentifier()); + + assertTrue(deleted.isPresent()); + assertEquals(createdAsset.getIdentifier(), deleted.get().getIdentifier()); + assertFalse(Files.exists(assetFilePath)); + assertTrue(assetManager.getAsset(createdAsset.getIdentifier()).isEmpty()); + } + + @Test + void testDeleteAssetReturnsEmptyForNonExistent() { + final Optional deleted = assetManager.deleteAsset("non-existent-id"); + + assertTrue(deleted.isEmpty()); + } + + @Test + void testInitializeCreatesStorageDirectory() { + final Path newStorageDir = tempDir.resolve("new-storage"); + assertFalse(Files.exists(newStorageDir)); + + final MockConnectorAssetManager newManager = new MockConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(newStorageDir); + newManager.initialize(context); + + assertTrue(Files.exists(newStorageDir)); + assertTrue(Files.isDirectory(newStorageDir)); + } + + private AssetManagerInitializationContext createInitializationContext(final Path storageDirectory) { + final Map properties = new HashMap<>(); + properties.put("directory", storageDirectory.toAbsolutePath().toString()); + + return new AssetManagerInitializationContext() { + @Override + public AssetReferenceLookup getAssetReferenceLookup() { + return null; + } + + @Override + public Map getProperties() { + return properties; + } + + @Override + public NodeTypeProvider getNodeTypeProvider() { + return null; + } + }; + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContextTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContextTest.java new file mode 100644 index 000000000000..659fe2e8fb73 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockNiFiConnectorWebContextTest.java @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.web.NiFiConnectorWebContext; +import org.apache.nifi.web.NiFiConnectorWebContext.ConnectorWebContext; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class MockNiFiConnectorWebContextTest { + + private static final String CONNECTOR_ID = "test-connector"; + + @Mock + private ConnectorRepository connectorRepository; + + @Mock + private ConnectorNode connectorNode; + + @Mock + private Connector connector; + + @Mock + private FrameworkFlowContext workingFlowContext; + + @Mock + private FrameworkFlowContext activeFlowContext; + + @Test + void testGetConnectorWebContextReturnsConnectorAndFlowContexts() { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + when(connectorNode.getConnector()).thenReturn(connector); + when(connectorNode.getWorkingFlowContext()).thenReturn(workingFlowContext); + when(connectorNode.getActiveFlowContext()).thenReturn(activeFlowContext); + + final NiFiConnectorWebContext context = new MockNiFiConnectorWebContext(connectorRepository); + final ConnectorWebContext result = context.getConnectorWebContext(CONNECTOR_ID); + + assertNotNull(result); + assertEquals(connector, result.connector()); + assertEquals(workingFlowContext, result.workingFlowContext()); + assertEquals(activeFlowContext, result.activeFlowContext()); + } + + @Test + void testGetConnectorWebContextThrowsForUnknownConnector() { + when(connectorRepository.getConnector("unknown-id")).thenReturn(null); + + final NiFiConnectorWebContext context = new MockNiFiConnectorWebContext(connectorRepository); + + assertThrows(IllegalArgumentException.class, () -> context.getConnectorWebContext("unknown-id")); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockStateManagerProviderTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockStateManagerProviderTest.java new file mode 100644 index 000000000000..d8ecf8e8729d --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/MockStateManagerProviderTest.java @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNotSame; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MockStateManagerProviderTest { + + @Test + void testGetStateManagerReturnsSameInstance() { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager1 = provider.getStateManager("component1", false); + final StateManager stateManager2 = provider.getStateManager("component1", false); + + assertNotNull(stateManager1); + assertSame(stateManager1, stateManager2); + } + + @Test + void testGetStateManagerReturnsDifferentInstancesForDifferentComponents() { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager1 = provider.getStateManager("component1", false); + final StateManager stateManager2 = provider.getStateManager("component2", false); + + assertNotNull(stateManager1); + assertNotNull(stateManager2); + assertNotSame(stateManager1, stateManager2); + } + + @Test + void testOnComponentRemoved() { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager1 = provider.getStateManager("component1", false); + assertNotNull(stateManager1); + + provider.onComponentRemoved("component1"); + + final StateManager stateManager2 = provider.getStateManager("component1", false); + assertNotNull(stateManager2); + assertNotSame(stateManager1, stateManager2); + } + + @Test + void testShutdown() { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + provider.getStateManager("component1", false); + provider.getStateManager("component2", false); + + provider.shutdown(); + + final StateManager stateManager = provider.getStateManager("component1", false); + assertNotNull(stateManager); + } + + @Test + void testClusterProviderEnabled() { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + assertFalse(provider.isClusterProviderEnabled()); + + provider.enableClusterProvider(); + assertTrue(provider.isClusterProviderEnabled()); + + provider.disableClusterProvider(); + assertFalse(provider.isClusterProviderEnabled()); + } + + @Test + void testStateManagerSetAndGetLocalState() throws IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + + final Map state = new HashMap<>(); + state.put("key1", "value1"); + state.put("key2", "value2"); + + stateManager.setState(state, Scope.LOCAL); + + final StateMap stateMap = stateManager.getState(Scope.LOCAL); + assertNotNull(stateMap); + assertTrue(stateMap.getStateVersion().isPresent()); + assertEquals("value1", stateMap.get("key1")); + assertEquals("value2", stateMap.get("key2")); + assertEquals(2, stateMap.toMap().size()); + } + + @Test + void testStateManagerSetAndGetClusterState() throws IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + + final Map state = new HashMap<>(); + state.put("key1", "value1"); + state.put("key2", "value2"); + + stateManager.setState(state, Scope.CLUSTER); + + final StateMap stateMap = stateManager.getState(Scope.CLUSTER); + assertNotNull(stateMap); + assertTrue(stateMap.getStateVersion().isPresent()); + assertEquals("value1", stateMap.get("key1")); + assertEquals("value2", stateMap.get("key2")); + } + + @Test + void testStateManagerReplace() throws IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + + final Map initialState = new HashMap<>(); + initialState.put("key1", "value1"); + stateManager.setState(initialState, Scope.LOCAL); + + final StateMap oldStateMap = stateManager.getState(Scope.LOCAL); + + final Map newState = new HashMap<>(); + newState.put("key1", "value2"); + + final boolean replaced = stateManager.replace(oldStateMap, newState, Scope.LOCAL); + assertTrue(replaced); + + final StateMap updatedStateMap = stateManager.getState(Scope.LOCAL); + assertEquals("value2", updatedStateMap.get("key1")); + + final boolean replacedAgain = stateManager.replace(oldStateMap, initialState, Scope.LOCAL); + assertFalse(replacedAgain); + } + + @Test + void testStateManagerClear() throws IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + + final Map state = new HashMap<>(); + state.put("key1", "value1"); + stateManager.setState(state, Scope.LOCAL); + + stateManager.clear(Scope.LOCAL); + + final StateMap stateMap = stateManager.getState(Scope.LOCAL); + assertNotNull(stateMap); + assertTrue(stateMap.toMap().isEmpty()); + } + + @Test + void testStateKeyDropSupported() { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager1 = provider.getStateManager("component1", false); + assertFalse(stateManager1.isStateKeyDropSupported()); + + final StateManager stateManager2 = provider.getStateManager("component2", true); + assertTrue(stateManager2.isStateKeyDropSupported()); + } + + @Test + void testThreadSafety() throws InterruptedException, IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + final int threadCount = 10; + final int operationsPerThread = 100; + final ExecutorService executorService = Executors.newFixedThreadPool(threadCount); + final CountDownLatch startLatch = new CountDownLatch(1); + final CountDownLatch completionLatch = new CountDownLatch(threadCount); + final AtomicInteger successCount = new AtomicInteger(0); + + for (int i = 0; i < threadCount; i++) { + final int threadId = i; + executorService.submit(() -> { + try { + startLatch.await(); + for (int j = 0; j < operationsPerThread; j++) { + final Map state = new HashMap<>(); + state.put("thread", String.valueOf(threadId)); + state.put("operation", String.valueOf(j)); + stateManager.setState(state, Scope.LOCAL); + + final StateMap stateMap = stateManager.getState(Scope.LOCAL); + if (stateMap != null && stateMap.toMap().size() == 2) { + successCount.incrementAndGet(); + } + } + } catch (final Exception e) { + e.printStackTrace(); + } finally { + completionLatch.countDown(); + } + }); + } + + startLatch.countDown(); + completionLatch.await(30, TimeUnit.SECONDS); + executorService.shutdown(); + + assertEquals(threadCount * operationsPerThread, successCount.get()); + final StateMap finalState = stateManager.getState(Scope.LOCAL); + assertNotNull(finalState); + assertEquals(2, finalState.toMap().size()); + } + + @Test + void testEmptyStateMap() throws IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + + final StateMap stateMap = stateManager.getState(Scope.LOCAL); + assertNotNull(stateMap); + assertFalse(stateMap.getStateVersion().isPresent()); + assertTrue(stateMap.toMap().isEmpty()); + } + + @Test + void testVersionIncrementing() throws IOException { + final MockStateManagerProvider provider = new MockStateManagerProvider(); + final StateManager stateManager = provider.getStateManager("component1", false); + + final Map state1 = new HashMap<>(); + state1.put("key", "value1"); + stateManager.setState(state1, Scope.LOCAL); + final StateMap stateMap1 = stateManager.getState(Scope.LOCAL); + + final Map state2 = new HashMap<>(); + state2.put("key", "value2"); + stateManager.setState(state2, Scope.LOCAL); + final StateMap stateMap2 = stateManager.getState(Scope.LOCAL); + + assertTrue(stateMap1.getStateVersion().isPresent()); + assertTrue(stateMap2.getStateVersion().isPresent()); + + final long version1 = Long.parseLong(stateMap1.getStateVersion().get()); + final long version2 = Long.parseLong(stateMap2.getStateVersion().get()); + assertTrue(version2 > version1); + } +} + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServerJettyTest.java b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServerJettyTest.java new file mode 100644 index 000000000000..980d8341c975 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-server/src/test/java/org/apache/nifi/mock/connector/server/StandardConnectorMockServerJettyTest.java @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector.server; + +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.bundle.BundleDetails; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; +import java.util.Set; +import java.util.jar.JarEntry; +import java.util.jar.JarOutputStream; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class StandardConnectorMockServerJettyTest { + + private static final String NAR_DEPENDENCIES_PATH = "NAR-INF/bundled-dependencies"; + + @TempDir + private Path tempDir; + + @Test + void testFindWarsDiscoversSingleWarFile() throws Exception { + final Path bundleWorkingDir = tempDir.resolve("test-bundle"); + final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH); + Files.createDirectories(depsDir); + + final Path warFile = depsDir.resolve("my-app.war"); + createConnectorWar(warFile); + + final Bundle bundle = createBundle(bundleWorkingDir); + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of(bundle)); + + assertEquals(1, wars.size()); + assertTrue(wars.containsKey(warFile.toFile())); + assertEquals(bundle, wars.get(warFile.toFile())); + } + + @Test + void testFindWarsIgnoresNonWarFiles() throws Exception { + final Path bundleWorkingDir = tempDir.resolve("test-bundle"); + final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH); + Files.createDirectories(depsDir); + + Files.createFile(depsDir.resolve("some-lib.jar")); + Files.createFile(depsDir.resolve("config.xml")); + + final Bundle bundle = createBundle(bundleWorkingDir); + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of(bundle)); + + assertTrue(wars.isEmpty()); + } + + @Test + void testFindWarsIgnoresWarWithoutConnectorManifest() throws Exception { + final Path bundleWorkingDir = tempDir.resolve("test-bundle"); + final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH); + Files.createDirectories(depsDir); + + final Path warFile = depsDir.resolve("non-connector.war"); + createWarWithoutConnectorManifest(warFile); + + final Bundle bundle = createBundle(bundleWorkingDir); + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of(bundle)); + + assertTrue(wars.isEmpty()); + } + + @Test + void testFindWarsHandlesMissingDependenciesDirectory() throws Exception { + final Path bundleWorkingDir = tempDir.resolve("empty-bundle"); + Files.createDirectories(bundleWorkingDir); + + final Bundle bundle = createBundle(bundleWorkingDir); + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of(bundle)); + + assertTrue(wars.isEmpty()); + } + + @Test + void testFindWarsDiscoversMultipleWarFiles() throws Exception { + final Path bundleWorkingDir = tempDir.resolve("multi-war-bundle"); + final Path depsDir = bundleWorkingDir.resolve(NAR_DEPENDENCIES_PATH); + Files.createDirectories(depsDir); + + createConnectorWar(depsDir.resolve("app-one.war")); + createConnectorWar(depsDir.resolve("app-two.war")); + Files.createFile(depsDir.resolve("some-lib.jar")); + + final Bundle bundle = createBundle(bundleWorkingDir); + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of(bundle)); + + assertEquals(2, wars.size()); + } + + @Test + void testFindWarsFromMultipleBundles() throws Exception { + final Path bundleDir1 = tempDir.resolve("bundle-1"); + final Path depsDir1 = bundleDir1.resolve(NAR_DEPENDENCIES_PATH); + Files.createDirectories(depsDir1); + createConnectorWar(depsDir1.resolve("first-app.war")); + + final Path bundleDir2 = tempDir.resolve("bundle-2"); + final Path depsDir2 = bundleDir2.resolve(NAR_DEPENDENCIES_PATH); + Files.createDirectories(depsDir2); + createConnectorWar(depsDir2.resolve("second-app.war")); + + final Bundle bundle1 = createBundle(bundleDir1, "test-bundle-1"); + final Bundle bundle2 = createBundle(bundleDir2, "test-bundle-2"); + + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of(bundle1, bundle2)); + + assertEquals(2, wars.size()); + } + + @Test + void testGetHttpPortReturnsNegativeOneWhenNoServer() { + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + assertEquals(-1, server.getHttpPort()); + } + + @Test + void testFindWarsReturnsEmptyForEmptyBundleSet() throws Exception { + final StandardConnectorMockServer server = new StandardConnectorMockServer(); + final Map wars = server.findWars(Set.of()); + assertTrue(wars.isEmpty()); + } + + private static void createConnectorWar(final Path warPath) throws IOException { + try (final JarOutputStream jarOut = new JarOutputStream(new FileOutputStream(warPath.toFile()))) { + jarOut.putNextEntry(new JarEntry("META-INF/nifi-connector")); + jarOut.closeEntry(); + } + } + + private static void createWarWithoutConnectorManifest(final Path warPath) throws IOException { + try (final JarOutputStream jarOut = new JarOutputStream(new FileOutputStream(warPath.toFile()))) { + jarOut.putNextEntry(new JarEntry("WEB-INF/web.xml")); + jarOut.closeEntry(); + } + } + + private Bundle createBundle(final Path workingDir) { + return createBundle(workingDir, "test-bundle"); + } + + private Bundle createBundle(final Path workingDir, final String artifactId) { + final BundleDetails details = new BundleDetails.Builder() + .workingDir(workingDir.toFile()) + .coordinate(new BundleCoordinate("org.test", artifactId, "1.0.0")) + .build(); + + return new Bundle(details, ClassLoader.getSystemClassLoader()); + } + +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/pom.xml new file mode 100644 index 000000000000..d227ec9e68a8 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/pom.xml @@ -0,0 +1,130 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-test-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-integration-tests + jar + + + + org.slf4j + slf4j-api + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-connector-mock + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-connector-mock-api + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-lookup-service-api + 2.9.0-SNAPSHOT + provided + + + + + org.junit.jupiter + junit-jupiter + test + + + + + org.apache.nifi + nifi-connector-mock-server-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-connector-mock-test-connectors-nar + 2.9.0-SNAPSHOT + nar + + + + + org.apache.nifi + nifi-standard-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-standard-services-api-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-standard-shared-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-update-attribute-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-lookup-services-nar + 2.9.0-SNAPSHOT + nar + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + copy-nar + package + + copy-dependencies + + + ${project.build.directory}/libDir + jar,nar + compile + + + + + + + + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/AllowableValuesIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/AllowableValuesIT.java new file mode 100644 index 000000000000..a350730b722a --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/AllowableValuesIT.java @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors.tests; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.mock.connector.StandardConnectorTestRunner; +import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Integration tests that verify fetchable allowable values are properly validated + * during configuration verification. These tests exercise the fix for NIFI-15258, + * where {@code StandardConnectorNode.fetchAllowableValues} incorrectly used the + * active flow context instead of the working flow context passed as a parameter. + * + *

The {@code AllowableValuesConnector} returns allowable values only when the + * flow context type is WORKING. If the framework incorrectly passes the ACTIVE + * context, the connector returns an empty list, causing all values to be accepted.

+ */ +public class AllowableValuesIT { + + private static final String CONNECTOR_CLASS = "org.apache.nifi.mock.connectors.AllowableValuesConnector"; + + @Test + public void testVerifyConfigurationRejectsInvalidAllowableValue() throws IOException { + try (final ConnectorTestRunner runner = createRunner()) { + final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Selection", Map.of("Color", "purple")); + + final List failedResults = result.getFailedResults(); + assertFalse(failedResults.isEmpty(), "Expected at least one failed result for invalid allowable value 'purple'"); + assertEquals(1, failedResults.size()); + + final String explanation = failedResults.getFirst().getExplanation(); + assertTrue(explanation.contains("allowable values"), "Expected explanation to mention allowable values but was: " + explanation); + } + } + + @Test + public void testVerifyConfigurationAcceptsValidAllowableValue() throws IOException { + try (final ConnectorTestRunner runner = createRunner()) { + final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Selection", Map.of("Color", "red")); + result.assertNoFailures(); + } + } + + @Test + public void testVerifyConfigurationAcceptsAllValidAllowableValues() throws IOException { + try (final ConnectorTestRunner runner = createRunner()) { + for (final String validColor : List.of("red", "green", "blue")) { + final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Selection", Map.of("Color", validColor)); + result.assertNoFailures(); + } + } + } + + private ConnectorTestRunner createRunner() { + return new StandardConnectorTestRunner.Builder() + .connectorClassName(CONNECTOR_CLASS) + .narLibraryDirectory(new File("target/libDir")) + .build(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CreateConnectorIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CreateConnectorIT.java new file mode 100644 index 000000000000..18884a8d8309 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CreateConnectorIT.java @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors.tests; + +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.mock.connector.StandardConnectorTestRunner; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.util.List; +import java.util.Optional; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class CreateConnectorIT { + + @Test + public void testCreateStartAndStopGenerateAndUpdateConnector() throws IOException { + try (final ConnectorTestRunner testRunner = new StandardConnectorTestRunner.Builder() + .connectorClassName("org.apache.nifi.mock.connectors.GenerateAndLog") + .narLibraryDirectory(new File("target/libDir")) + .build()) { + + // Verify the active flow snapshot reflects the initial flow loaded from Generate_and_Update.json + final VersionedExternalFlow activeFlow = testRunner.getActiveFlowSnapshot(); + final VersionedProcessGroup rootGroup = activeFlow.getFlowContents(); + + final Set processors = rootGroup.getProcessors(); + assertEquals(3, processors.size()); + assertTrue(findProcessorByType(processors, "org.apache.nifi.processors.standard.GenerateFlowFile").isPresent()); + assertTrue(findProcessorByType(processors, "org.apache.nifi.processors.standard.LookupAttribute").isPresent()); + assertTrue(findProcessorByType(processors, "org.apache.nifi.processors.attributes.UpdateAttribute").isPresent()); + + assertEquals(2, rootGroup.getConnections().size()); + + final Set controllerServices = rootGroup.getControllerServices(); + assertEquals(1, controllerServices.size()); + assertEquals("org.apache.nifi.lookup.SimpleKeyValueLookupService", controllerServices.iterator().next().getType()); + + testRunner.startConnector(); + testRunner.stopConnector(); + } + } + + @Test + public void testConnectorWithMissingBundleFailsValidate() throws IOException { + + try (final ConnectorTestRunner testRunner = new StandardConnectorTestRunner.Builder() + .connectorClassName("org.apache.nifi.mock.connectors.MissingBundleConnector") + .narLibraryDirectory(new File("target/libDir")) + .build()) { + + final List results = testRunner.validate(); + assertEquals(results.size(), 1); + final String message = results.getFirst().getExplanation(); + assertTrue(message.contains("com.example.nonexistent:missing-nar:1.0.0"), "Expected exception message to contain missing bundle coordinates but was: " + message); + assertTrue(message.contains("com.example.nonexistent.MissingProcessor"), "Expected exception message to contain missing processor type but was: " + message); + } + } + + private Optional findProcessorByType(final Set processors, final String type) { + for (final VersionedProcessor processor : processors) { + if (type.equals(processor.getType())) { + return Optional.of(processor); + } + } + return Optional.empty(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CronScheduleIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CronScheduleIT.java new file mode 100644 index 000000000000..10dea97a4327 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/CronScheduleIT.java @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors.tests; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.mock.connector.StandardConnectorTestRunner; +import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertFalse; + +/** + * Integration tests that verify CRON expression validation during configuration verification. + * The {@code CronScheduleConnector} maps the "Trigger Schedule" connector property to a + * CRON-driven GenerateFlowFile processor's scheduling period parameter, and verification + * delegates to the processor's validation which checks the CRON expression. + */ +public class CronScheduleIT { + + private static final String CONNECTOR_CLASS = "org.apache.nifi.mock.connectors.CronScheduleConnector"; + + @Test + public void testValidCronExpression() throws IOException { + try (final ConnectorTestRunner runner = createRunner()) { + final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Schedule", + Map.of("Trigger Schedule", "0 0 * * * *")); + result.assertNoFailures(); + } + } + + @Test + public void testInvalidCronExpression() throws IOException { + try (final ConnectorTestRunner runner = createRunner()) { + final ConnectorConfigVerificationResult result = runner.verifyConfiguration("Schedule", + Map.of("Trigger Schedule", "invalid-cron")); + final List failedResults = result.getFailedResults(); + assertFalse(failedResults.isEmpty()); + } + } + + private ConnectorTestRunner createRunner() { + return new StandardConnectorTestRunner.Builder() + .connectorClassName(CONNECTOR_CLASS) + .narLibraryDirectory(new File("target/libDir")) + .build(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockControllerServiceIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockControllerServiceIT.java new file mode 100644 index 000000000000..1f044ccf21c6 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockControllerServiceIT.java @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors.tests; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.Validator; +import org.apache.nifi.controller.AbstractControllerService; +import org.apache.nifi.lookup.LookupFailureException; +import org.apache.nifi.lookup.StringLookupService; +import org.apache.nifi.mock.connector.StandardConnectorTestRunner; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.io.File; +import java.io.IOException; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.atomic.AtomicLong; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class MockControllerServiceIT { + + @Test + @Timeout(10) + public void testMockControllerService() throws IOException { + try (final ConnectorTestRunner runner = new StandardConnectorTestRunner.Builder() + .narLibraryDirectory(new File("target/libDir")) + .connectorClassName("org.apache.nifi.mock.connectors.GenerateAndLog") + .mockControllerService("org.apache.nifi.lookup.SimpleKeyValueLookupService", MockStringLookupService.class) + .build()) { + + runner.startConnector(); + + // Wait until MockStringLookupService.lookup is invoked at least once. + // We use @Timeout on the test to avoid hanging indefinitely in case of failure. + while (MockStringLookupService.lookupCounter.get() < 1) { + Thread.yield(); + } + + assertTrue(MockStringLookupService.lookupCounter.get() >= 1); + + runner.stopConnector(); + } + } + + public static class MockStringLookupService extends AbstractControllerService implements StringLookupService { + private static final AtomicLong lookupCounter = new AtomicLong(0L); + + @Override + protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName) { + return new PropertyDescriptor.Builder() + .name(propertyDescriptorName) + .addValidator(Validator.VALID) + .dynamic(true) + .build(); + } + + @Override + public Optional lookup(final Map coordinates) throws LookupFailureException { + lookupCounter.incrementAndGet(); + return Optional.of("mock-value"); + } + + @Override + public Set getRequiredKeys() { + return Set.of("key"); + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockProcessorIT.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockProcessorIT.java new file mode 100644 index 000000000000..465d0340002b --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-integration-tests/src/test/java/org/apache/nifi/mock/connectors/tests/MockProcessorIT.java @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors.tests; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.mock.connector.StandardConnectorTestRunner; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.io.File; +import java.io.IOException; +import java.util.Set; +import java.util.concurrent.atomic.AtomicLong; + +public class MockProcessorIT { + + @Test + @Timeout(10) + public void testMockProcessor() throws IOException { + try (final ConnectorTestRunner runner = new StandardConnectorTestRunner.Builder() + .narLibraryDirectory(new File("target/libDir")) + .connectorClassName("org.apache.nifi.mock.connectors.GenerateAndLog") + .mockProcessor("org.apache.nifi.processors.attributes.UpdateAttribute", MockProcessor.class) + .build()) { + + runner.startConnector(); + + // Wait until MockProcessor is triggered at least once. We use @Timeout on the test to avoid + // hanging indefinitely in case of failure. + while (MockProcessor.invocationCounter.get() < 1) { + Thread.yield(); + } + runner.stopConnector(); + } + } + + public static class MockProcessor extends AbstractProcessor { + private static final AtomicLong invocationCounter = new AtomicLong(0L); + + public static final Relationship REL_SUCCESS = new Relationship.Builder() + .name("success") + .build(); + + @Override + public Set getRelationships() { + return Set.of(REL_SUCCESS); + } + + // Support any properties so that properties of UpdateAttribute can be set without the Processor becoming invalid + @Override + protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName) { + return new PropertyDescriptor.Builder() + .name(propertyDescriptorName) + .addValidator(Validator.VALID) + .build(); + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + invocationCounter.incrementAndGet(); + + final FlowFile flowFile = session.get(); + if (flowFile != null) { + session.transfer(flowFile, REL_SUCCESS); + } + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors-nar/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors-nar/pom.xml new file mode 100644 index 000000000000..c2d17f8b288a --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors-nar/pom.xml @@ -0,0 +1,35 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-test-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-test-connectors-nar + nar + + + + org.apache.nifi + nifi-connector-mock-test-connectors + 2.9.0-SNAPSHOT + + + + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/pom.xml new file mode 100644 index 000000000000..479fa9db0784 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/pom.xml @@ -0,0 +1,48 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-test-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-test-connectors + jar + + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-connector-utils + 2.9.0-SNAPSHOT + + + + + + org.junit.jupiter + junit-jupiter + test + + + + + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/AllowableValuesConnector.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/AllowableValuesConnector.java new file mode 100644 index 000000000000..a263482eda28 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/AllowableValuesConnector.java @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors; + +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.FlowContextType; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; + +/** + * A test connector with a configuration step that has fetchable allowable values. + * The {@link #fetchAllowableValues} method returns allowable values only when invoked + * with a WORKING {@link FlowContextType}. If invoked with an ACTIVE context, it returns + * an empty list. This design verifies that the framework passes the correct (working) + * context during configuration verification, reproducing the scenario fixed in NIFI-15258. + */ +public class AllowableValuesConnector extends AbstractConnector { + + static final ConnectorPropertyDescriptor COLOR = new ConnectorPropertyDescriptor.Builder() + .name("Color") + .description("The color to select") + .allowableValuesFetchable(true) + .required(true) + .build(); + + static final ConnectorPropertyGroup SELECTION_GROUP = new ConnectorPropertyGroup.Builder() + .name("Color Selection") + .addProperty(COLOR) + .build(); + + static final ConfigurationStep SELECTION_STEP = new ConfigurationStep.Builder() + .name("Selection") + .propertyGroups(List.of(SELECTION_GROUP)) + .build(); + + private static final List CONFIGURATION_STEPS = List.of(SELECTION_STEP); + + @Override + public VersionedExternalFlow getInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource("flows/Generate_and_Update.json"); + } + + @Override + public List getConfigurationSteps() { + return CONFIGURATION_STEPS; + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext workingContext) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext) { + if ("Selection".equals(stepName) && "Color".equals(propertyName)) { + if (flowContext.getType() == FlowContextType.WORKING) { + return List.of( + new AllowableValue("red", "Red"), + new AllowableValue("green", "Green"), + new AllowableValue("blue", "Blue") + ); + } + + // Return empty when the context is ACTIVE, simulating the pre-fix behavior + // where using the wrong context caused allowable values to be empty and + // any value to be incorrectly accepted. + return List.of(); + } + + return super.fetchAllowableValues(stepName, propertyName, flowContext); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/CronScheduleConnector.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/CronScheduleConnector.java new file mode 100644 index 000000000000..1e257c245c6d --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/CronScheduleConnector.java @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; + +/** + * A test connector that validates CRON scheduling expressions via the "Trigger Schedule" parameter. + * The underlying flow ({@code Cron_Schedule_Connector.json}) contains a CRON-driven + * GenerateFlowFile processor whose scheduling period references the {@code #{Trigger Schedule}} parameter. + * Verification delegates to the processor's validate/verify cycle, which checks the CRON expression. + */ +public class CronScheduleConnector extends AbstractConnector { + + static final String SCHEDULE_STEP_NAME = "Schedule"; + static final String TRIGGER_SCHEDULE_PARAM = "Trigger Schedule"; + + static final ConnectorPropertyDescriptor TRIGGER_SCHEDULE = new ConnectorPropertyDescriptor.Builder() + .name(TRIGGER_SCHEDULE_PARAM) + .description("CRON expression for the GenerateFlowFile trigger schedule") + .required(true) + .build(); + + static final ConnectorPropertyGroup SCHEDULE_GROUP = new ConnectorPropertyGroup.Builder() + .name("Schedule Settings") + .addProperty(TRIGGER_SCHEDULE) + .build(); + + static final ConfigurationStep SCHEDULE_STEP = new ConfigurationStep.Builder() + .name(SCHEDULE_STEP_NAME) + .propertyGroups(List.of(SCHEDULE_GROUP)) + .build(); + + @Override + public VersionedExternalFlow getInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource("flows/Cron_Schedule_Connector.json"); + } + + @Override + public List getConfigurationSteps() { + return List.of(SCHEDULE_STEP); + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext flowContext) throws FlowUpdateException { + if (SCHEDULE_STEP_NAME.equals(stepName)) { + final String triggerSchedule = flowContext.getConfigurationContext() + .getProperty(SCHEDULE_STEP_NAME, TRIGGER_SCHEDULE_PARAM).getValue(); + final VersionedExternalFlow flow = getInitialFlow(); + VersionedFlowUtils.setParameterValue(flow, TRIGGER_SCHEDULE_PARAM, triggerSchedule); + getInitializationContext().updateFlow(flowContext, flow); + } + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + if (SCHEDULE_STEP_NAME.equals(stepName)) { + final ConnectorConfigurationContext configContext = flowContext.getConfigurationContext().createWithOverrides(stepName, overrides); + final String triggerSchedule = configContext.getProperty(SCHEDULE_STEP_NAME, TRIGGER_SCHEDULE_PARAM).getValue(); + + final VersionedExternalFlow flow = getInitialFlow(); + VersionedFlowUtils.setParameterValue(flow, TRIGGER_SCHEDULE_PARAM, triggerSchedule); + + final ProcessorFacade generateFlowFile = flowContext.getRootGroup().getProcessors().stream() + .filter(p -> p.getDefinition().getType().endsWith("GenerateFlowFile")) + .findFirst() + .orElseThrow(); + + return generateFlowFile.verify(flow, Map.of()); + } + + return List.of(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/GenerateAndLog.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/GenerateAndLog.java new file mode 100644 index 000000000000..96b9fa63c38e --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/GenerateAndLog.java @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; + +public class GenerateAndLog extends AbstractConnector { + + @Override + public VersionedExternalFlow getInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource("flows/Generate_and_Update.json"); + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext workingContext) { + return List.of(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/MissingBundleConnector.java b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/MissingBundleConnector.java new file mode 100644 index 000000000000..d49674ffd038 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/java/org/apache/nifi/mock/connectors/MissingBundleConnector.java @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connectors; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** + * A test connector that returns an initial flow containing a processor with a bundle that does not exist. + * This is used to test the behavior when a connector's initial flow references unavailable components. + */ +public class MissingBundleConnector extends AbstractConnector { + + @Override + public VersionedExternalFlow getInitialFlow() { + final VersionedProcessGroup rootGroup = new VersionedProcessGroup(); + rootGroup.setIdentifier(UUID.randomUUID().toString()); + rootGroup.setInstanceIdentifier(UUID.randomUUID().toString()); + rootGroup.setName("Missing Bundle Connector Flow"); + rootGroup.setPosition(new Position(0.0, 0.0)); + rootGroup.setProcessGroups(new HashSet<>()); + rootGroup.setConnections(new HashSet<>()); + rootGroup.setInputPorts(new HashSet<>()); + rootGroup.setOutputPorts(new HashSet<>()); + rootGroup.setControllerServices(new HashSet<>()); + rootGroup.setFunnels(new HashSet<>()); + rootGroup.setLabels(new HashSet<>()); + + final VersionedProcessor missingProcessor = new VersionedProcessor(); + missingProcessor.setIdentifier(UUID.randomUUID().toString()); + missingProcessor.setInstanceIdentifier(UUID.randomUUID().toString()); + missingProcessor.setName("Missing Processor"); + missingProcessor.setType("com.example.nonexistent.MissingProcessor"); + missingProcessor.setPosition(new Position(100.0, 100.0)); + missingProcessor.setScheduledState(ScheduledState.ENABLED); + missingProcessor.setSchedulingPeriod("0 sec"); + missingProcessor.setSchedulingStrategy("TIMER_DRIVEN"); + missingProcessor.setExecutionNode("ALL"); + missingProcessor.setPenaltyDuration("30 sec"); + missingProcessor.setYieldDuration("1 sec"); + missingProcessor.setBulletinLevel("WARN"); + missingProcessor.setRunDurationMillis(0L); + missingProcessor.setConcurrentlySchedulableTaskCount(1); + missingProcessor.setAutoTerminatedRelationships(new HashSet<>()); + missingProcessor.setProperties(Map.of()); + missingProcessor.setPropertyDescriptors(Map.of()); + missingProcessor.setGroupIdentifier(rootGroup.getIdentifier()); + + final Bundle missingBundle = new Bundle(); + missingBundle.setGroup("com.example.nonexistent"); + missingBundle.setArtifact("missing-nar"); + missingBundle.setVersion("1.0.0"); + missingProcessor.setBundle(missingBundle); + + rootGroup.setProcessors(new HashSet<>(List.of(missingProcessor))); + + final VersionedExternalFlow externalFlow = new VersionedExternalFlow(); + externalFlow.setFlowContents(rootGroup); + return externalFlow; + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext workingContext) { + return List.of(); + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector new file mode 100644 index 000000000000..379d9026452a --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.mock.connectors.AllowableValuesConnector +org.apache.nifi.mock.connectors.GenerateAndLog +org.apache.nifi.mock.connectors.MissingBundleConnector +org.apache.nifi.mock.connectors.CronScheduleConnector \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Cron_Schedule_Connector.json b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Cron_Schedule_Connector.json new file mode 100644 index 000000000000..d078d4d6fa83 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Cron_Schedule_Connector.json @@ -0,0 +1,410 @@ +{ + "flowContents": { + "identifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "instanceIdentifier": "4f8ca484-019c-1000-955f-68c5defeb22b", + "name": "Cron_Schedule_Connector", + "comments": "", + "position": { + "x": -1254.0, + "y": -437.70139741897583 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a", + "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9", + "name": "UpdateAttribute", + "comments": "", + "position": { + "x": -360.0, + "y": 24.0 + }, + "type": "org.apache.nifi.processors.attributes.UpdateAttribute", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-update-attribute-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "Delete Attributes Expression": null, + "Store State": "Do not store state", + "Cache Value Lookup Cache Size": "100", + "Stateful Variables Initial Value": null + }, + "propertyDescriptors": { + "Delete Attributes Expression": { + "name": "Delete Attributes Expression", + "displayName": "Delete Attributes Expression", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Store State": { + "name": "Store State", + "displayName": "Store State", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Cache Value Lookup Cache Size": { + "name": "Cache Value Lookup Cache Size", + "displayName": "Cache Value Lookup Cache Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Stateful Variables Initial Value": { + "name": "Stateful Variables Initial Value", + "displayName": "Stateful Variables Initial Value", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 25, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "success" + ], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + }, + { + "identifier": "54ab8572-0e10-39a5-b553-931f9c253023", + "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78", + "name": "GenerateFlowFile", + "comments": "", + "position": { + "x": -360.0, + "y": -371.5 + }, + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "File Size": "0B", + "Batch Size": "1", + "Unique FlowFiles": "false", + "Mime Type": null, + "Custom Text": null, + "Character Set": "UTF-8", + "Data Format": "Text", + "key": "test.key" + }, + "propertyDescriptors": { + "File Size": { + "name": "File Size", + "displayName": "File Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Batch Size": { + "name": "Batch Size", + "displayName": "Batch Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Unique FlowFiles": { + "name": "Unique FlowFiles", + "displayName": "Unique FlowFiles", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Mime Type": { + "name": "Mime Type", + "displayName": "Mime Type", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Custom Text": { + "name": "Custom Text", + "displayName": "Custom Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Character Set": { + "name": "Character Set", + "displayName": "Character Set", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Data Format": { + "name": "Data Format", + "displayName": "Data Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "key": { + "name": "key", + "displayName": "key", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": true + } + }, + "style": {}, + "schedulingPeriod": "#{Trigger Schedule}", + "schedulingStrategy": "CRON_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + }, + { + "identifier": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a", + "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0", + "name": "LookupAttribute", + "comments": "", + "position": { + "x": -360.0, + "y": -168.0 + }, + "type": "org.apache.nifi.processors.standard.LookupAttribute", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "Lookup Service": "b013f870-aee8-3cc4-b022-ac385ded928d", + "test.attribute": "${key}", + "Include Empty Values": "true" + }, + "propertyDescriptors": { + "Lookup Service": { + "name": "Lookup Service", + "displayName": "Lookup Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "test.attribute": { + "name": "test.attribute", + "displayName": "test.attribute", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": true + }, + "Include Empty Values": { + "name": "Include Empty Values", + "displayName": "Include Empty Values", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "failure" + ], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + } + ], + "inputPorts": [], + "outputPorts": [], + "connections": [ + { + "identifier": "893ad2c9-4a07-3447-b772-6b7149cfd6c1", + "instanceIdentifier": "5b9c96a4-7982-3746-2937-45511fb20c96", + "name": "", + "source": { + "id": "54ab8572-0e10-39a5-b553-931f9c253023", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "GenerateFlowFile", + "comments": "", + "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78" + }, + "destination": { + "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "LookupAttribute", + "comments": "", + "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + }, + { + "identifier": "f63459de-cf1f-3773-8f93-405518e085e4", + "instanceIdentifier": "4f93c5e2-019c-1000-2bea-06d8d611d5f6", + "name": "", + "source": { + "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "LookupAttribute", + "comments": "", + "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0" + }, + "destination": { + "id": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "UpdateAttribute", + "comments": "", + "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "matched", + "unmatched" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [ + { + "identifier": "b013f870-aee8-3cc4-b022-ac385ded928d", + "instanceIdentifier": "4f8d53d4-019c-1000-d376-abfaa091dc37", + "name": "SimpleKeyValueLookupService", + "comments": "", + "type": "org.apache.nifi.lookup.SimpleKeyValueLookupService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-lookup-services-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "test.key": "Test Value" + }, + "propertyDescriptors": { + "test.key": { + "name": "test.key", + "displayName": "test.key", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": true + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.lookup.StringLookupService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2026.1.20.21-SNAPSHOT" + } + }, + { + "type": "org.apache.nifi.lookup.LookupService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2026.1.20.21-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + } + ], + "parameterContextName": "Generate Parameters", + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP" + }, + "externalControllerServices": {}, + "parameterContexts": { + "Generate Parameters": { + "name": "Generate Parameters", + "parameters": [ + { + "name": "Trigger Schedule", + "description": "", + "sensitive": false, + "provided": false, + "value": "" + } + ], + "inheritedParameterContexts": [], + "description": "", + "componentType": "PARAMETER_CONTEXT" + } + }, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Generate_and_Update.json b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Generate_and_Update.json new file mode 100644 index 000000000000..d4f5fc798ede --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/nifi-connector-mock-test-connectors/src/main/resources/flows/Generate_and_Update.json @@ -0,0 +1,393 @@ +{ + "flowContents": { + "identifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "instanceIdentifier": "4f8ca484-019c-1000-955f-68c5defeb22b", + "name": "Generate_and_Update", + "comments": "", + "position": { + "x": -1254.0, + "y": -437.70139741897583 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a", + "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9", + "name": "UpdateAttribute", + "comments": "", + "position": { + "x": -360.0, + "y": 24.0 + }, + "type": "org.apache.nifi.processors.attributes.UpdateAttribute", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-update-attribute-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "Delete Attributes Expression": null, + "Store State": "Do not store state", + "Cache Value Lookup Cache Size": "100", + "Stateful Variables Initial Value": null + }, + "propertyDescriptors": { + "Delete Attributes Expression": { + "name": "Delete Attributes Expression", + "displayName": "Delete Attributes Expression", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Store State": { + "name": "Store State", + "displayName": "Store State", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Cache Value Lookup Cache Size": { + "name": "Cache Value Lookup Cache Size", + "displayName": "Cache Value Lookup Cache Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Stateful Variables Initial Value": { + "name": "Stateful Variables Initial Value", + "displayName": "Stateful Variables Initial Value", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 25, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "success" + ], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + }, + { + "identifier": "54ab8572-0e10-39a5-b553-931f9c253023", + "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78", + "name": "GenerateFlowFile", + "comments": "", + "position": { + "x": -360.0, + "y": -371.5 + }, + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "File Size": "0B", + "Batch Size": "1", + "Unique FlowFiles": "false", + "Mime Type": null, + "Custom Text": null, + "Character Set": "UTF-8", + "Data Format": "Text", + "key": "test.key" + }, + "propertyDescriptors": { + "File Size": { + "name": "File Size", + "displayName": "File Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Batch Size": { + "name": "Batch Size", + "displayName": "Batch Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Unique FlowFiles": { + "name": "Unique FlowFiles", + "displayName": "Unique FlowFiles", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Mime Type": { + "name": "Mime Type", + "displayName": "Mime Type", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Custom Text": { + "name": "Custom Text", + "displayName": "Custom Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Character Set": { + "name": "Character Set", + "displayName": "Character Set", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Data Format": { + "name": "Data Format", + "displayName": "Data Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "key": { + "name": "key", + "displayName": "key", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": true + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + }, + { + "identifier": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a", + "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0", + "name": "LookupAttribute", + "comments": "", + "position": { + "x": -360.0, + "y": -168.0 + }, + "type": "org.apache.nifi.processors.standard.LookupAttribute", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "Lookup Service": "b013f870-aee8-3cc4-b022-ac385ded928d", + "test.attribute": "${key}", + "Include Empty Values": "true" + }, + "propertyDescriptors": { + "Lookup Service": { + "name": "Lookup Service", + "displayName": "Lookup Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "test.attribute": { + "name": "test.attribute", + "displayName": "test.attribute", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": true + }, + "Include Empty Values": { + "name": "Include Empty Values", + "displayName": "Include Empty Values", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "failure" + ], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + } + ], + "inputPorts": [], + "outputPorts": [], + "connections": [ + { + "identifier": "893ad2c9-4a07-3447-b772-6b7149cfd6c1", + "instanceIdentifier": "5b9c96a4-7982-3746-2937-45511fb20c96", + "name": "", + "source": { + "id": "54ab8572-0e10-39a5-b553-931f9c253023", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "GenerateFlowFile", + "comments": "", + "instanceIdentifier": "6d6b9cd3-6d31-330a-40f9-185959ad1c78" + }, + "destination": { + "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "LookupAttribute", + "comments": "", + "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + }, + { + "identifier": "f63459de-cf1f-3773-8f93-405518e085e4", + "instanceIdentifier": "4f93c5e2-019c-1000-2bea-06d8d611d5f6", + "name": "", + "source": { + "id": "34c1ac1b-1f21-3fd6-a734-726d5b142b7a", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "LookupAttribute", + "comments": "", + "instanceIdentifier": "4f8d0670-019c-1000-1ac6-c81ef75a70d0" + }, + "destination": { + "id": "4d4160b1-736c-3be4-bc3a-84fc6eb4ad2a", + "type": "PROCESSOR", + "groupId": "1800c04e-f9b9-3293-bfc7-b35f43e0706c", + "name": "UpdateAttribute", + "comments": "", + "instanceIdentifier": "499f9781-71c5-367f-aa17-5441bff29de9" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "matched", + "unmatched" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [ + { + "identifier": "b013f870-aee8-3cc4-b022-ac385ded928d", + "instanceIdentifier": "4f8d53d4-019c-1000-d376-abfaa091dc37", + "name": "SimpleKeyValueLookupService", + "comments": "", + "type": "org.apache.nifi.lookup.SimpleKeyValueLookupService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-lookup-services-nar", + "version": "2026.1.20.21-SNAPSHOT" + }, + "properties": { + "test.key": "Test Value" + }, + "propertyDescriptors": { + "test.key": { + "name": "test.key", + "displayName": "test.key", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": true + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.lookup.LookupService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2026.1.20.21-SNAPSHOT" + } + }, + { + "type": "org.apache.nifi.lookup.StringLookupService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2026.1.20.21-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "1800c04e-f9b9-3293-bfc7-b35f43e0706c" + } + ], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP" + }, + "externalControllerServices": {}, + "parameterContexts": {}, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/pom.xml new file mode 100644 index 000000000000..bc7d2ebc1a8a --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock-test-bundle/pom.xml @@ -0,0 +1,33 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock-test-bundle + pom + + + nifi-connector-mock-test-connectors + nifi-connector-mock-test-connectors-nar + nifi-connector-mock-integration-tests + + + diff --git a/nifi-connector-mock-bundle/nifi-connector-mock/pom.xml b/nifi-connector-mock-bundle/nifi-connector-mock/pom.xml new file mode 100644 index 000000000000..ef7d8adb538e --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock/pom.xml @@ -0,0 +1,147 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-connector-mock-bundle + 2.9.0-SNAPSHOT + + + nifi-connector-mock + jar + + + + org.slf4j + slf4j-api + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-nar-utils + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-properties + 2.9.0-SNAPSHOT + provided + + + org.apache.nifi + nifi-connector-mock-api + 2.9.0-SNAPSHOT + + + + + org.apache.nifi + nifi-framework-nar-utils + 2.9.0-SNAPSHOT + + + + + + org.apache.nifi + nifi-framework-api + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-property-utils + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-python-framework-api + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-stateless-api + 2.9.0-SNAPSHOT + + + + + org.slf4j + log4j-over-slf4j + + + org.slf4j + jul-to-slf4j + + + org.slf4j + jcl-over-slf4j + + + + + org.junit.jupiter + junit-jupiter + test + + + + + org.apache.nifi + nifi-framework-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-jetty-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-provenance-repository-nar + 2.9.0-SNAPSHOT + nar + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + copy-nar + package + + copy-dependencies + + + ${project.build.directory}/libDir + jar,nar + compile + + + + + + + \ No newline at end of file diff --git a/nifi-connector-mock-bundle/nifi-connector-mock/src/main/java/org/apache/nifi/mock/connector/StandardConnectorTestRunner.java b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/java/org/apache/nifi/mock/connector/StandardConnectorTestRunner.java new file mode 100644 index 000000000000..7153f8d868b3 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/java/org/apache/nifi/mock/connector/StandardConnectorTestRunner.java @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.mock.connector; + +import org.apache.nifi.NiFiServer; +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.controller.ControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult; +import org.apache.nifi.mock.connector.server.ConnectorMockServer; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.apache.nifi.nar.ExtensionMapping; +import org.apache.nifi.nar.NarClassLoaders; +import org.apache.nifi.nar.NarUnpackMode; +import org.apache.nifi.nar.NarUnpacker; +import org.apache.nifi.nar.SystemBundle; +import org.apache.nifi.processor.Processor; +import org.apache.nifi.util.NiFiProperties; + +import java.io.Closeable; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Path; +import java.time.Duration; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.Set; + +public class StandardConnectorTestRunner implements ConnectorTestRunner, Closeable { + private final File narLibraryDirectory; + private final int httpPort; + + private ConnectorMockServer mockServer; + + private StandardConnectorTestRunner(final Builder builder) { + this.narLibraryDirectory = builder.narLibraryDirectory; + this.httpPort = builder.httpPort; + + try { + bootstrapInstance(); + } catch (final Exception e) { + throw new RuntimeException("Failed to bootstrap ConnectorTestRunner", e); + } + + // It is important that we register the processor and controller service mocks before instantiating the connector. + // Otherwise, the call to instantiateConnector will initialize the Connector, which may update the flow. + // If the flow is updated before the mocks are registered, the components will be created without + // using the mocks. Subsequent updates to the flow will not replace the components already created because + // these are not recognized as updates to the flow, since the framework assumes that the type of a component + // with a given ID does not change. + builder.processorMocks.forEach(mockServer::mockProcessor); + builder.controllerServiceMocks.forEach(mockServer::mockControllerService); + + mockServer.instantiateConnector(builder.connectorClassName); + } + + private void bootstrapInstance() throws IOException, ClassNotFoundException { + final List libDirectoryPaths = List.of(narLibraryDirectory.toPath()); + final File extensionsWorkingDir = new File("target/work/extensions"); + final File frameworkWorkingDir = new File("target/work/framework"); + + final Bundle systemBundle = SystemBundle.create(narLibraryDirectory.getAbsolutePath(), ClassLoader.getSystemClassLoader()); + + final ExtensionMapping extensionMapping = NarUnpacker.unpackNars(systemBundle, frameworkWorkingDir, extensionsWorkingDir, libDirectoryPaths, true, + NarClassLoaders.FRAMEWORK_NAR_ID, true, false, NarUnpackMode.UNPACK_INDIVIDUAL_JARS, bundleCoordinate -> true); + + final NarClassLoaders narClassLoaders = new NarClassLoaders(); + narClassLoaders.init(frameworkWorkingDir, extensionsWorkingDir); + + final NiFiServer nifiServer = narClassLoaders.getServer(); + if (nifiServer == null) { + throw new RuntimeException("Could not find NiFiServer instance"); + } + if (!(nifiServer instanceof ConnectorMockServer)) { + throw new RuntimeException("Test ClassPath does not contain ConnectorMockServer. " + + "Ensure that the appropriate module is packaged in the NAR library directory: " + narLibraryDirectory.getAbsolutePath()); + } + + // Set Application Server Class Loader for subsequent operations + final ClassLoader applicationServerClassLoader = narClassLoaders.getServer().getClass().getClassLoader(); + Thread.currentThread().setContextClassLoader(applicationServerClassLoader); + + final Set narBundles = narClassLoaders.getBundles(); + + final Properties additionalProperties = new Properties(); + if (httpPort >= 0) { + additionalProperties.setProperty(NiFiProperties.WEB_HTTP_PORT, String.valueOf(httpPort)); + } + + final NiFiProperties properties; + try (final InputStream propertiesIn = getClass().getClassLoader().getResourceAsStream("nifi.properties")) { + properties = NiFiProperties.createBasicNiFiProperties(propertiesIn, additionalProperties); + } + + nifiServer.initialize(properties, systemBundle, narBundles, extensionMapping); + nifiServer.start(); + + mockServer = (ConnectorMockServer) nifiServer; + mockServer.registerMockBundle(getClass().getClassLoader(), new File(extensionsWorkingDir, "mock-implementations-bundle")); + } + + @Override + public void close() { + if (mockServer != null) { + mockServer.stop(); + } + } + + @Override + public void applyUpdate() throws FlowUpdateException { + mockServer.applyUpdate(); + } + + @Override + public void configure(final String stepName, final StepConfiguration configuration) throws FlowUpdateException { + mockServer.configure(stepName, configuration); + } + + @Override + public void configure(final String stepName, final Map propertyValues) throws FlowUpdateException { + mockServer.configure(stepName, propertyValues); + } + + @Override + public void configure(final String stepName, final Map propertyValues, final Map propertyReferences) throws FlowUpdateException { + mockServer.configure(stepName, propertyValues, propertyReferences); + } + + @Override + public SecretReference createSecretReference(final String secretName) { + return mockServer.createSecretReference(secretName); + } + + @Override + public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map propertyValueOverrides) { + return mockServer.verifyConfiguration(stepName, propertyValueOverrides); + } + + @Override + public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final Map propertyValueOverrides, + final Map referenceOverrides) { + + return mockServer.verifyConfiguration(stepName, propertyValueOverrides, referenceOverrides); + } + + @Override + public ConnectorConfigVerificationResult verifyConfiguration(final String stepName, final StepConfiguration configurationOverrides) { + return mockServer.verifyConfiguration(stepName, configurationOverrides); + } + + @Override + public void addSecret(final String name, final String value) { + mockServer.addSecret(name, value); + } + + @Override + public AssetReference addAsset(final File file) { + return mockServer.addAsset(file); + } + + @Override + public AssetReference addAsset(final String assetName, final InputStream contents) { + return mockServer.addAsset(assetName, contents); + } + + @Override + public void startConnector() { + mockServer.startConnector(); + } + + @Override + public void stopConnector() { + mockServer.stopConnector(); + } + + @Override + public void waitForDataIngested(final Duration maxWaitTime) { + mockServer.waitForDataIngested(maxWaitTime); + } + + @Override + public void waitForIdle(final Duration maxWaitTime) { + mockServer.waitForIdle(maxWaitTime); + } + + @Override + public void waitForIdle(final Duration minIdleTime, final Duration maxWaitTime) { + mockServer.waitForIdle(minIdleTime, maxWaitTime); + } + + @Override + public List validate() { + return mockServer.validate(); + } + + @Override + public int getHttpPort() { + return mockServer.getHttpPort(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName) { + return mockServer.fetchAllowableValues(stepName, propertyName); + } + + @Override + public VersionedExternalFlow getActiveFlowSnapshot() { + return mockServer.getActiveFlowSnapshot(); + } + + @Override + public VersionedExternalFlow getWorkingFlowSnapshot() { + return mockServer.getWorkingFlowSnapshot(); + } + + + public static class Builder { + private String connectorClassName; + private File narLibraryDirectory; + private int httpPort = -1; + private final Map> processorMocks = new HashMap<>(); + private final Map> controllerServiceMocks = new HashMap<>(); + + public Builder connectorClassName(final String connectorClassName) { + this.connectorClassName = connectorClassName; + return this; + } + + public Builder narLibraryDirectory(final File libDirectory) { + this.narLibraryDirectory = libDirectory; + return this; + } + + public Builder httpPort(final int httpPort) { + this.httpPort = httpPort; + return this; + } + + public Builder mockProcessor(final String processorType, final Class mockProcessorClass) { + processorMocks.put(processorType, mockProcessorClass); + return this; + } + + public Builder mockControllerService(final String controllerServiceType, final Class mockControllerServiceClass) { + controllerServiceMocks.put(controllerServiceType, mockControllerServiceClass); + return this; + } + + public StandardConnectorTestRunner build() { + if (!narLibraryDirectory.exists() || !narLibraryDirectory.isDirectory()) { + throw new IllegalArgumentException("NAR file does not exist or is not a directory: " + narLibraryDirectory.getAbsolutePath()); + } + + return new StandardConnectorTestRunner(this); + } + } +} diff --git a/nifi-connector-mock-bundle/nifi-connector-mock/src/main/resources/nifi.properties b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/resources/nifi.properties new file mode 100644 index 000000000000..44deabc71ad3 --- /dev/null +++ b/nifi-connector-mock-bundle/nifi-connector-mock/src/main/resources/nifi.properties @@ -0,0 +1,165 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Core Properties # +nifi.flow.configuration.file=target/nifi-storage/conf/flow.json.gz +nifi.flow.configuration.archive.enabled=true +nifi.flow.configuration.archive.dir=target/nifi-storage/conf/archive/ +nifi.flow.configuration.archive.max.time=30 days +nifi.flow.configuration.archive.max.storage=500 MB +nifi.flow.configuration.archive.max.count= +nifi.flowcontroller.autoResumeState=true +nifi.flowcontroller.graceful.shutdown.period=10 sec +nifi.flowservice.writedelay.interval=500 ms +nifi.administrative.yield.duration=3 sec +# If a component has no work to do (is "bored"), how long should we wait before checking again for work? +nifi.bored.yield.duration=10 millis +nifi.queue.backpressure.count=10000 +nifi.queue.backpressure.size=1 GB + +nifi.nar.library.directory=./lib +nifi.nar.library.autoload.directory=./extensions +nifi.nar.working.directory=./work/nar/ + +##################### +# Python Extensions # +##################### +# Uncomment in order to enable Python Extensions. +#nifi.python.command=python +nifi.python.framework.source.directory=./python/framework +nifi.python.extensions.source.directory.default=./python/extensions +nifi.python.working.directory=./work/python +nifi.python.max.processes=100 +nifi.python.max.processes.per.extension.type=10 + +#################### +# State Management # +#################### +nifi.state.management.configuration.file=target/nifi-storage/state-management.xml +# The ID of the local state provider +nifi.state.management.provider.local=local-provider + +# Database Settings +nifi.database.directory=target/nifi-storage/database_repository + +# FlowFile Repository +nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository +nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog +nifi.flowfile.repository.directory=target/nifi-storage/flowfile_repository +nifi.flowfile.repository.checkpoint.interval=1 mins +nifi.flowfile.repository.always.sync=false + +nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager +nifi.queue.swap.threshold=20000 +nifi.swap.in.period=5 sec +nifi.swap.in.threads=1 +nifi.swap.out.period=5 sec +nifi.swap.out.threads=4 + +# Content Repository +nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository +nifi.content.claim.max.appendable.size=50 KB +nifi.content.repository.directory.default=target/nifi-storage/content_repository +nifi.content.repository.archive.max.retention.period=12 hours +nifi.content.repository.archive.max.usage.percentage=90% +nifi.content.repository.archive.enabled=false +nifi.content.repository.always.sync=false + +# Provenance Repository Properties +nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository + +# Volatile Provenance Respository Properties +nifi.provenance.repository.buffer.size=1000 + +# Component Status Repository +nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository +nifi.components.status.repository.buffer.size=1440 +nifi.components.status.snapshot.frequency=1 min + +# NAR Persistence Provider Properties +nifi.nar.persistence.provider.properties.directory=target/nifi-storage/nar_repository + +# Asset Management +nifi.asset.manager.properties.directory=target/nifi-storage/assets + +# Connector Asset Manager +nifi.connector.asset.manager.implementation=org.apache.nifi.mock.connector.server.MockConnectorAssetManager + +# Secrets Manager +nifi.secrets.manager.implementation=org.apache.nifi.mock.connector.server.secrets.ConnectorTestRunnerSecretsManager + +# Site to Site properties +nifi.remote.input.host=localhost +nifi.remote.input.secure=true +nifi.remote.input.socket.port=7780 +nifi.remote.input.http.enabled=true +nifi.remote.input.http.transaction.ttl=30 sec +nifi.remote.contents.cache.expiration=30 secs + +# web properties # +nifi.web.war.directory=./lib +nifi.web.http.host= +nifi.web.http.port= +nifi.web.http.network.interface.default= +nifi.web.https.host=localhost +nifi.web.https.port=0 +nifi.web.https.network.interface.default= +nifi.web.jetty.working.directory=target/work/jetty +nifi.web.jetty.threads=200 +nifi.web.max.header.size=16 KB +nifi.web.proxy.context.path= +nifi.web.proxy.host= + +# security properties # +nifi.sensitive.props.key=nifi-connector-mock-tests +nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256 + +nifi.security.keystore= +nifi.security.keystoreType=PKCS12 +nifi.security.keystorePasswd= +nifi.security.keyPasswd= +nifi.security.truststore= +nifi.security.truststoreType=PKCS12 +nifi.security.truststorePasswd= +nifi.security.user.authorizer=system-test-authorizer +nifi.security.user.login.identity.provider= + +# cluster common properties (all nodes must have same values) # +nifi.cluster.protocol.heartbeat.interval=5 sec +nifi.cluster.protocol.is.secure=true + +# cluster node properties (only configure for cluster nodes) # +nifi.cluster.is.node=false +nifi.cluster.node.address= +nifi.cluster.node.protocol.port= +nifi.cluster.node.protocol.threads=10 +nifi.cluster.node.protocol.max.threads=50 +nifi.cluster.node.event.history.size=25 +nifi.cluster.node.connection.timeout=5 sec +nifi.cluster.node.read.timeout=5 sec +nifi.cluster.node.max.concurrent.requests=100 +nifi.cluster.firewall.file= +nifi.cluster.flow.election.max.wait.time=5 mins +nifi.cluster.flow.election.max.candidates= + +# cluster load balancing properties # +nifi.cluster.load.balance.host= +nifi.cluster.load.balance.port=6342 +nifi.cluster.load.balance.connections.per.node=4 +nifi.cluster.load.balance.max.thread.count=8 +nifi.cluster.load.balance.comms.timeout=30 sec + +# Connector Repository +nifi.components.connectors.repository.implementation=org.apache.nifi.mock.connector.server.MockConnectorRepository diff --git a/nifi-connector-mock-bundle/pom.xml b/nifi-connector-mock-bundle/pom.xml new file mode 100644 index 000000000000..64fd76bf604e --- /dev/null +++ b/nifi-connector-mock-bundle/pom.xml @@ -0,0 +1,34 @@ + + + + 4.0.0 + + org.apache.nifi + nifi + 2.9.0-SNAPSHOT + + + nifi-connector-mock-bundle + pom + + + nifi-connector-mock + nifi-connector-mock-api + nifi-connector-mock-server + nifi-connector-mock-server-nar + nifi-connector-mock-test-bundle + + \ No newline at end of file diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/pom.xml b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/pom.xml new file mode 100644 index 000000000000..a58046afaf9a --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/pom.xml @@ -0,0 +1,40 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-kafka-to-s3-bundle + 2.9.0-SNAPSHOT + + + nifi-kafka-to-s3-connector + + + + org.slf4j + slf4j-api + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-connector-utils + + + diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaConnectionStep.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaConnectionStep.java new file mode 100644 index 000000000000..793b1e9be0de --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaConnectionStep.java @@ -0,0 +1,135 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.kafkas3; + +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; + +public class KafkaConnectionStep { + public static final String STEP_NAME = "Kafka Connection"; + + public static final ConnectorPropertyDescriptor KAFKA_BROKERS = new ConnectorPropertyDescriptor.Builder() + .name("Kafka Brokers") + .description("A comma-separated list of Kafka brokers to connect to.") + .required(true) + .validators(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR) + .type(PropertyType.STRING_LIST) + .build(); + + public static final ConnectorPropertyDescriptor SECURITY_PROTOCOL = new ConnectorPropertyDescriptor.Builder() + .name("Security Protocol") + .description("The security protocol to use when connecting to Kafka brokers.") + .required(true) + .type(PropertyType.STRING) + .defaultValue("SASL_PLAINTEXT") + .allowableValues("PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL") + .build(); + + public static final ConnectorPropertyDescriptor SASL_MECHANISM = new ConnectorPropertyDescriptor.Builder() + .name("SASL Mechanism") + .description("The SASL mechanism to use for authentication.") + .required(true) + .type(PropertyType.STRING) + .allowableValues("PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512", "GSSAPI", "OAUTHBEARER") + .dependsOn(SECURITY_PROTOCOL, "SASL_PLAINTEXT", "SASL_SSL") + .defaultValue("SCRAM-SHA-512") + .build(); + + public static final ConnectorPropertyDescriptor USERNAME = new ConnectorPropertyDescriptor.Builder() + .name("Username") + .description("The username for SASL authentication.") + .required(true) + .type(PropertyType.STRING) + .validators(StandardValidators.NON_EMPTY_VALIDATOR) + .dependsOn(SECURITY_PROTOCOL, "SASL_PLAINTEXT", "SASL_SSL") + .build(); + + public static final ConnectorPropertyDescriptor PASSWORD = new ConnectorPropertyDescriptor.Builder() + .name("Password") + .description("The password for SASL authentication.") + .required(true) + .type(PropertyType.SECRET) + .validators(StandardValidators.NON_EMPTY_VALIDATOR) + .dependsOn(SECURITY_PROTOCOL, "SASL_PLAINTEXT", "SASL_SSL") + .build(); + + public static final ConnectorPropertyGroup KAFKA_SERVER_GROUP = new ConnectorPropertyGroup.Builder() + .name("Kafka Server Settings") + .description("Settings for connecting to the Kafka server") + .properties(List.of( + KAFKA_BROKERS, + SECURITY_PROTOCOL, + SASL_MECHANISM, + USERNAME, + PASSWORD + )) + .build(); + + public static final ConnectorPropertyDescriptor SCHEMA_REGISTRY_URL = new ConnectorPropertyDescriptor.Builder() + .name("Schema Registry URL") + .description("The URL of the Schema Registry.") + .required(false) + .type(PropertyType.STRING) + .validators(StandardValidators.URL_VALIDATOR) + .validators(Validator.VALID) + .build(); + + public static final ConnectorPropertyDescriptor SCHEMA_REGISTRY_USERNAME = new ConnectorPropertyDescriptor.Builder() + .name("Schema Registry Username") + .description("The username for Schema Registry authentication.") + .required(false) + .type(PropertyType.STRING) + .validators(StandardValidators.NON_EMPTY_VALIDATOR) + .dependsOn(SCHEMA_REGISTRY_URL) + .build(); + + public static final ConnectorPropertyDescriptor SCHEMA_REGISTRY_PASSWORD = new ConnectorPropertyDescriptor.Builder() + .name("Schema Registry Password") + .description("The password for Schema Registry authentication.") + .required(false) + .type(PropertyType.SECRET) + .validators(StandardValidators.NON_EMPTY_VALIDATOR) + .dependsOn(SCHEMA_REGISTRY_URL) + .build(); + + + public static final ConnectorPropertyGroup SCHEMA_REGISTRY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Schema Registry Settings") + .description("Settings for connecting to the Schema Registry") + .properties(List.of( + SCHEMA_REGISTRY_URL, + SCHEMA_REGISTRY_USERNAME, + SCHEMA_REGISTRY_PASSWORD + )) + .build(); + + public static final ConfigurationStep KAFKA_CONNECTION_STEP = new ConfigurationStep.Builder() + .name(STEP_NAME) + .description("Configure Kafka connection settings") + .propertyGroups(List.of( + KAFKA_SERVER_GROUP, + SCHEMA_REGISTRY_GROUP + )) + .build(); +} diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3.java new file mode 100644 index 000000000000..9c4bc1b7d386 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3.java @@ -0,0 +1,297 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.kafkas3; + +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ControllerServiceFacade; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.stream.Collectors; + +@CapabilityDescription("Provides the ability to ingest data from Apache Kafka topics, merge it together into an object of reasonable " + + "size, and write that data to Amazon S3.") +@Tags({"kafka", "s3"}) +public class KafkaToS3 extends AbstractConnector { + + private static final List configurationSteps = List.of( + KafkaConnectionStep.KAFKA_CONNECTION_STEP, + KafkaTopicsStep.KAFKA_TOPICS_STEP, + S3Step.S3_STEP + ); + + private volatile CompletableFuture drainFlowFileFuture = null; + + + @Override + public List getConfigurationSteps() { + return configurationSteps; + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final String activeS3DataFormat = activeContext.getConfigurationContext().getProperty( + S3Step.S3_STEP_NAME, S3Step.S3_DATA_FORMAT.getName()).getValue(); + final String workingS3DataFormat = workingContext.getConfigurationContext().getProperty( + S3Step.S3_STEP_NAME, S3Step.S3_DATA_FORMAT.getName()).getValue(); + + if (!activeS3DataFormat.equals(workingS3DataFormat)) { + getLogger().info("S3 Data Format changed from {} to {}; draining flow before updating it", activeS3DataFormat, workingS3DataFormat); + + drainFlowFileFuture = drainFlowFiles(activeContext); + try { + drainFlowFileFuture.get(5, TimeUnit.MINUTES); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new FlowUpdateException("Interrupted while waiting for all FlowFiles to drain from the flow", e); + } catch (final TimeoutException e) { + throw new FlowUpdateException("Timed out waiting for all FlowFiles to drain from the flow", e); + } catch (final ExecutionException e) { + throw new FlowUpdateException("Failed to drain FlowFiles from flow", e.getCause()); + } + + getLogger().info("All FlowFiles drained from the flow; proceeding with flow update"); + } + } + + @Override + public void abortUpdate(final FlowContext workingContext, final Throwable throwable) { + if (drainFlowFileFuture != null) { + drainFlowFileFuture.cancel(true); + drainFlowFileFuture = null; + } + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final VersionedExternalFlow flow = buildFlow(workingContext.getConfigurationContext()); + getInitializationContext().updateFlow(activeContext, flow); + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return KafkaToS3FlowBuilder.loadInitialFlow(); + } + + @Override + public void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + final VersionedExternalFlow flow = buildFlow(workingContext.getConfigurationContext()); + getInitializationContext().updateFlow(workingContext, flow); + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext workingFlowContext) { + // Get the current ConfigurationContext and then create a new one that contains the provided property values + final ConnectorConfigurationContext configurationContext = workingFlowContext.getConfigurationContext().createWithOverrides(stepName, propertyValueOverrides); + final VersionedExternalFlow flow = buildFlow(configurationContext); + + // Validate Connectivity + if (stepName.equals(KafkaConnectionStep.STEP_NAME)) { + return verifyKafkaConnectivity(workingFlowContext, flow); + } + if (stepName.equals(KafkaTopicsStep.STEP_NAME)) { + final List results = new ArrayList<>(); + results.addAll(verifyTopicsExists(workingFlowContext)); + results.addAll(verifyKafkaParsability(workingFlowContext, flow)); + return results; + } + + return Collections.emptyList(); + } + + private List verifyKafkaParsability(final FlowContext workingFlowContext, final VersionedExternalFlow flow) { + // Enable Controller Services necessary for parsing records. + // We determine which Controller Services are referenced by the flow and enable them, but we do not use + // getRootGroup().getLifecycle().enableReferencedControllerServices(ControllerServiceReferenceScope.INCLUDE_REFERENCED_SERVICES_ONLY) + // because that would include the Controller Services that are referenced by the currently configured flow, and it's possible that the + // what is being verified uses a different set of Controller Services (e.g., the verified flow may use a JSON Reader while the current + // flow uses an Avro Reader). + final ProcessGroupFacade rootGroup = workingFlowContext.getRootGroup(); + final Set referencedServices = VersionedFlowUtils.getReferencedControllerServices(flow.getFlowContents()); + final Set serviceIds = referencedServices.stream() + .map(VersionedControllerService::getIdentifier) + .collect(Collectors.toSet()); + + try { + rootGroup.getLifecycle().enableControllerServices(serviceIds).get(10, TimeUnit.SECONDS); + } catch (final Exception e) { + return List.of(new ConfigVerificationResult.Builder() + .verificationStepName("Record Parsing") + .outcome(Outcome.FAILED) + .explanation("Failed to enable Controller Services due to " + e) + .build()); + } + + try { + final ProcessorFacade consumeKafkaFacade = findProcessors(rootGroup, + processor -> processor.getDefinition().getType().endsWith("ConsumeKafka")).getFirst(); + + final List configVerificationResults = consumeKafkaFacade.verify(flow, Map.of()); + for (final ConfigVerificationResult result : configVerificationResults) { + if (result.getOutcome() == Outcome.FAILED) { + return List.of(result); + } + } + + return Collections.emptyList(); + } finally { + rootGroup.getLifecycle().disableControllerServices(serviceIds); + } + } + + + private List verifyTopicsExists(final FlowContext workingFlowContext) { + final List topicsAvailable; + try { + topicsAvailable = getAvailableTopics(workingFlowContext); + } catch (final Exception e) { + return List.of(new ConfigVerificationResult.Builder() + .verificationStepName("Verify Kafka topics exist") + .outcome(Outcome.SKIPPED) + .explanation("Unable to validate that topics exist due to " + e) + .build()); + } + + final Set topicNames = new HashSet<>(topicsAvailable); + final List specifiedTopics = workingFlowContext.getConfigurationContext().getProperty(KafkaTopicsStep.STEP_NAME, + KafkaTopicsStep.TOPIC_NAMES.getName()).asList(); + final String missingTopics = specifiedTopics.stream() + .filter(topic -> !topicNames.contains(topic)) + .collect(Collectors.joining(", ")); + + if (!missingTopics.isEmpty()) { + return List.of(new ConfigVerificationResult.Builder() + .verificationStepName("Verify Kafka topics exist") + .outcome(Outcome.FAILED) + .explanation("The following topics do not exist in the Kafka cluster: " + missingTopics) + .build()); + } else { + return List.of(new ConfigVerificationResult.Builder() + .verificationStepName("Verify Kafka topics exist") + .outcome(Outcome.SUCCESSFUL) + .explanation("All specified topics exist in the Kafka cluster") + .build()); + } + } + + private List verifyKafkaConnectivity(final FlowContext workingFlowContext, final VersionedExternalFlow flow) { + // Build a new version of the flow so that we can get the relevant properties of the Kafka Connection Service + final ControllerServiceFacade connectionService = getKafkaConnectionService(workingFlowContext); + final List configVerificationResults = connectionService.verify(flow, Map.of()); + + for (final ConfigVerificationResult result : configVerificationResults) { + if (result.getOutcome() == Outcome.FAILED) { + return List.of(new ConfigVerificationResult.Builder() + .verificationStepName("Verify Kafka connectivity") + .outcome(Outcome.FAILED) + .explanation(result.getExplanation()) + .build()); + } + } + + return Collections.emptyList(); + } + + private VersionedExternalFlow buildFlow(final ConnectorConfigurationContext configurationContext) { + final KafkaToS3FlowBuilder flowBuilder = new KafkaToS3FlowBuilder(configurationContext); + return flowBuilder.buildFlow(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext) { + if (stepName.equals(KafkaTopicsStep.STEP_NAME) && propertyName.equals(KafkaTopicsStep.TOPIC_NAMES.getName())) { + return createAllowableValues(getAvailableTopics(flowContext)); + } else if (stepName.equals(S3Step.S3_STEP_NAME) && propertyName.equals(S3Step.S3_REGION.getName())) { + return createAllowableValues(getPossibleS3Regions(flowContext)); + } + + return super.fetchAllowableValues(stepName, propertyName, flowContext); + } + + private List createAllowableValues(final List values) { + return values.stream().map(this::createAllowableValue).collect(Collectors.toList()); + } + + private DescribedValue createAllowableValue(final String value) { + return new AllowableValue(value, value, value); + } + + @SuppressWarnings("unchecked") + private List getAvailableTopics(final FlowContext flowContext) { + // If Kafka Brokers not yet set, return empty list + final ConnectorConfigurationContext config = flowContext.getConfigurationContext(); + if (!config.getProperty(KafkaConnectionStep.KAFKA_CONNECTION_STEP, KafkaConnectionStep.KAFKA_BROKERS).isSet()) { + return List.of(); + } + + final ControllerServiceFacade kafkaConnectionService = getKafkaConnectionService(flowContext); + + try { + return (List) kafkaConnectionService.invokeConnectorMethod("listTopicNames", Map.of()); + } catch (final Exception e) { + getLogger().warn("Failed to retrieve available Kafka topics", e); + return List.of(); + } + } + + private ControllerServiceFacade getKafkaConnectionService(final FlowContext flowContext) { + return flowContext.getRootGroup().getControllerServices().stream() + .filter(service -> service.getDefinition().getType().endsWith("Kafka3ConnectionService")) + .findFirst() + .orElseThrow(); + } + + @SuppressWarnings("unchecked") + private List getPossibleS3Regions(final FlowContext flowContext) { + final ProcessorFacade processorFacade = flowContext.getRootGroup().getProcessors().stream() + .filter(proc -> proc.getDefinition().getType().endsWith("PutS3Object")) + .findFirst() + .orElseThrow(); + + try { + return (List) processorFacade.invokeConnectorMethod("getAvailableRegions", Map.of()); + } catch (final InvocationFailedException e) { + getLogger().error("Failed to obtain list of available S3 regions", e); + return Collections.emptyList(); + } + } +} diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3FlowBuilder.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3FlowBuilder.java new file mode 100644 index 000000000000..e7d5a75c3338 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaToS3FlowBuilder.java @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.kafkas3; + +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.StepConfigurationContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.processor.DataUnit; + +import java.util.HashMap; +import java.util.Map; + +public class KafkaToS3FlowBuilder { + private static final String FLOW_JSON_PATH = "flows/Kafka_to_S3.json"; + + private final ConnectorConfigurationContext configContext; + + public KafkaToS3FlowBuilder(final ConnectorConfigurationContext configurationContext) { + this.configContext = configurationContext; + } + + public static VersionedExternalFlow loadInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource(FLOW_JSON_PATH); + } + + public VersionedExternalFlow buildFlow() { + final VersionedExternalFlow externalFlow = VersionedFlowUtils.loadFlowFromResource(FLOW_JSON_PATH); + configureSchemaRegistry(externalFlow); + + updateKafkaConnectionParameters(externalFlow); + updateSchemaRegistryParameters(externalFlow); + updateReaderWriter(externalFlow); + updateKafkaTopicsParameters(externalFlow); + updateS3Config(externalFlow); + + return externalFlow; + } + + private void configureSchemaRegistry(final VersionedExternalFlow externalFlow) { + final String schemaRegistryUrl = configContext.getProperty(KafkaConnectionStep.KAFKA_CONNECTION_STEP, KafkaConnectionStep.SCHEMA_REGISTRY_URL).getValue(); + + if (schemaRegistryUrl == null) { + final VersionedProcessGroup processGroup = externalFlow.getFlowContents(); + + // Remove any references to the Schema Registry service. + final VersionedControllerService schemaRegistryService = processGroup.getControllerServices().stream() + .filter(service -> service.getType().endsWith("ConfluentSchemaRegistry")) + .findFirst() + .orElseThrow(); + VersionedFlowUtils.removeControllerServiceReferences(processGroup, schemaRegistryService.getIdentifier()); + + final VersionedControllerService schemaReferenceReader = processGroup.getControllerServices().stream() + .filter(service -> service.getType().endsWith("ConfluentEncodedSchemaReferenceReader")) + .findFirst() + .orElseThrow(); + VersionedFlowUtils.removeControllerServiceReferences(processGroup, schemaReferenceReader.getIdentifier()); + + processGroup.getControllerServices().stream() + .filter(service -> service.getType().endsWith("JsonTreeReader")) + .forEach(service -> service.getProperties().put("Schema Access Strategy", "infer-schema")); + + processGroup.getControllerServices().stream() + .filter(service -> service.getType().endsWith("JsonRecordSetWriter")) + .forEach(service -> service.getProperties().put("Schema Write Strategy", "no-schema")); + } + } + + private void updateSchemaRegistryParameters(final VersionedExternalFlow externalFlow) { + final StepConfigurationContext stepContext = configContext.scopedToStep(KafkaConnectionStep.KAFKA_CONNECTION_STEP); + + final String schemaRegistryUrl = stepContext.getProperty(KafkaConnectionStep.SCHEMA_REGISTRY_URL).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry URLs", schemaRegistryUrl); + + if (schemaRegistryUrl == null) { + final Map properties = new HashMap<>(); + properties.put("schema-access-strategy", "infer-schema"); + properties.put("schema-registry", null); + properties.put("schema-reference-reader", null); + + externalFlow.getFlowContents().getControllerServices().stream() + .filter(service -> service.getType().endsWith("JsonTreeReader")) + .findFirst() + .ifPresent(service -> service.setProperties(properties)); + } else { + final String username = stepContext.getProperty(KafkaConnectionStep.SCHEMA_REGISTRY_USERNAME).getValue(); + final String password = stepContext.getProperty(KafkaConnectionStep.SCHEMA_REGISTRY_PASSWORD).getValue(); + + VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry Username", username); + VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry Password", password); + + final String authenticationType = (username == null || username.isEmpty()) ? "NONE" : "BASIC"; + VersionedFlowUtils.setParameterValue(externalFlow, "Schema Registry Authentication Type", authenticationType); + } + } + + private void updateKafkaConnectionParameters(final VersionedExternalFlow externalFlow) { + final StepConfigurationContext stepContext = configContext.scopedToStep(KafkaConnectionStep.KAFKA_CONNECTION_STEP); + + final String kafkaBrokers = stepContext.getProperty(KafkaConnectionStep.KAFKA_BROKERS).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Bootstrap Servers", kafkaBrokers); + + final String securityProtocol = stepContext.getProperty(KafkaConnectionStep.SECURITY_PROTOCOL).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Security Protocol", securityProtocol); + + if (securityProtocol.contains("SASL")) { + final String saslMechanism = stepContext.getProperty(KafkaConnectionStep.SASL_MECHANISM).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka SASL Mechanism", saslMechanism); + + final String username = stepContext.getProperty(KafkaConnectionStep.USERNAME).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka SASL Username", username); + + final String password = stepContext.getProperty(KafkaConnectionStep.PASSWORD).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka SASL Password", password); + } + } + + private void updateReaderWriter(final VersionedExternalFlow externalFlow) { + final VersionedProcessGroup rootGroup = externalFlow.getFlowContents(); + + final VersionedControllerService avroService = rootGroup.getControllerServices().stream() + .filter(service -> service.getType().endsWith("AvroReader")) + .findFirst() + .orElseThrow(); + + final String kafkaDataFormat = configContext.getProperty(KafkaTopicsStep.STEP_NAME, KafkaTopicsStep.KAFKA_DATA_FORMAT.getName()).getValue(); + if (!kafkaDataFormat.equalsIgnoreCase("JSON")) { + // Update ConsumeKafka processor to use Avro Reader + rootGroup.getProcessors().stream() + .filter(versionedProcessor -> versionedProcessor.getType().endsWith("ConsumeKafka")) + .findFirst() + .ifPresent(processor -> processor.getProperties().put("Record Reader", avroService.getIdentifier())); + } + + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Data Format", kafkaDataFormat); + + final String s3DataFormat = configContext.getProperty(S3Step.S3_STEP, S3Step.S3_DATA_FORMAT).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Data Format", s3DataFormat); + } + + private void updateKafkaTopicsParameters(final VersionedExternalFlow externalFlow) { + final StepConfigurationContext stepContext = configContext.scopedToStep(KafkaTopicsStep.STEP_NAME); + + final String topics = stepContext.getProperty(KafkaTopicsStep.TOPIC_NAMES.getName()).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Topic Names", topics); + + final String groupId = stepContext.getProperty(KafkaTopicsStep.CONSUMER_GROUP_ID.getName()).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Consumer Group ID", groupId); + + final String offsetReset = stepContext.getProperty(KafkaTopicsStep.OFFSET_RESET.getName()).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Kafka Auto Offset Reset", offsetReset); + } + + private void updateS3Config(final VersionedExternalFlow externalFlow) { + final StepConfigurationContext stepContext = configContext.scopedToStep(S3Step.S3_STEP); + + final String region = stepContext.getProperty(S3Step.S3_REGION).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Region", region); + + final String bucket = stepContext.getProperty(S3Step.S3_BUCKET).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Bucket", bucket); + + final String prefix = stepContext.getProperty(S3Step.S3_PREFIX).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Prefix", prefix); + + final String endpointOverrideUrl = stepContext.getProperty(S3Step.S3_ENDPOINT_OVERRIDE_URL).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Endpoint Override URL", endpointOverrideUrl); + + final String authStrategy = stepContext.getProperty(S3Step.S3_AUTHENTICATION_STRATEGY).getValue(); + if (authStrategy.equals(S3Step.DEFAULT_CREDENTIALS)) { + final VersionedControllerService credentialsService = externalFlow.getFlowContents().getControllerServices().stream() + .filter(service -> service.getType().endsWith("AWSCredentialsProviderControllerService")) + .findFirst() + .orElseThrow(); + + credentialsService.setProperties(Map.of("default-credentials", "true")); + } else { + final String accessKey = stepContext.getProperty(S3Step.S3_ACCESS_KEY_ID).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Access Key ID", accessKey); + + final String secretKey = stepContext.getProperty(S3Step.S3_SECRET_ACCESS_KEY).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "S3 Secret Access Key", secretKey); + } + + final long mergeBytes = stepContext.getProperty(S3Step.TARGET_OBJECT_SIZE).asDataSize(DataUnit.B).longValue(); + final String mergeSize = stepContext.getProperty(S3Step.TARGET_OBJECT_SIZE).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Target Object Size", mergeSize); + + // Max Bin size will be either 10% more than target size or target size + 100MB, whichever is smaller + final long maxBinSize = (long) Math.min(mergeBytes + 100_000_000, mergeBytes * 1.1D); + VersionedFlowUtils.setParameterValue(externalFlow, "Maximum Object Size", maxBinSize + " B"); + + final String mergeLatency = stepContext.getProperty(S3Step.MERGE_LATENCY).getValue(); + VersionedFlowUtils.setParameterValue(externalFlow, "Merge Latency", mergeLatency); + } +} diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaTopicsStep.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaTopicsStep.java new file mode 100644 index 000000000000..754a708c3955 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/KafkaTopicsStep.java @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.kafkas3; + +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; + +public class KafkaTopicsStep { + public static final String STEP_NAME = "Kafka Topics"; + + public static final ConnectorPropertyDescriptor TOPIC_NAMES = new ConnectorPropertyDescriptor.Builder() + .name("Topic Names") + .description("A comma-separated list of Kafka topics to consume from.") + .required(true) + .type(PropertyType.STRING_LIST) + .allowableValuesFetchable(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + public static final ConnectorPropertyDescriptor CONSUMER_GROUP_ID = new ConnectorPropertyDescriptor.Builder() + .name("Consumer Group ID") + .description("The consumer group ID to use when connecting to Kafka.") + .required(true) + .type(PropertyType.STRING) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + public static final ConnectorPropertyDescriptor OFFSET_RESET = new ConnectorPropertyDescriptor.Builder() + .name("Offset Reset") + .description("What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server.") + .required(true) + .type(PropertyType.STRING) + .defaultValue("latest") + .allowableValues("earliest", "latest", "none") + .build(); + + public static final ConnectorPropertyDescriptor KAFKA_DATA_FORMAT = new ConnectorPropertyDescriptor.Builder() + .name("Kafka Data Format") + .description("The format of the data in Kafka topics.") + .required(true) + .type(PropertyType.STRING) + .defaultValue("Avro") + .allowableValues("Avro", "JSON") + .build(); + + public static final ConnectorPropertyGroup KAFKA_TOPICS_GROUP = new ConnectorPropertyGroup.Builder() + .name("Kafka Topics Configuration") + .description("Properties for configuring Kafka topics consumption.") + .properties(List.of( + TOPIC_NAMES, + CONSUMER_GROUP_ID, + OFFSET_RESET, + KAFKA_DATA_FORMAT + )) + .build(); + + public static final ConfigurationStep KAFKA_TOPICS_STEP = new ConfigurationStep.Builder() + .name(STEP_NAME) + .description("Kafka topics to consume from.") + .propertyGroups(List.of( + KAFKA_TOPICS_GROUP + )) + .build(); + +} diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/S3Step.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/S3Step.java new file mode 100644 index 000000000000..ad29115c4ad6 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/java/org/apache/nifi/connectors/kafkas3/S3Step.java @@ -0,0 +1,156 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.kafkas3; + +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; + +public class S3Step { + public static final String S3_STEP_NAME = "S3 Configuration"; + public static final String ACCESS_KEY_ID_SECRET_KEY = "Access Key ID and Secret Key"; + public static final String DEFAULT_CREDENTIALS = "Default AWS Credentials"; + + + public static final ConnectorPropertyDescriptor S3_BUCKET = new ConnectorPropertyDescriptor.Builder() + .name("S3 Bucket") + .description("The name of the S3 bucket to write data to.") + .required(true) + .build(); + + public static final ConnectorPropertyDescriptor S3_REGION = new ConnectorPropertyDescriptor.Builder() + .name("S3 Region") + .description("The AWS region where the S3 bucket is located.") + .allowableValuesFetchable(true) + .required(true) + .build(); + + public static final ConnectorPropertyDescriptor S3_DATA_FORMAT = new ConnectorPropertyDescriptor.Builder() + .name("S3 Data Format") + .description("The format to use when writing data to S3.") + .required(true) + .defaultValue("JSON") + .allowableValues("Avro", "JSON") + .build(); + + public static final ConnectorPropertyDescriptor S3_PREFIX = new ConnectorPropertyDescriptor.Builder() + .name("S3 Prefix") + .description("An optional prefix to prepend to all object keys written to the S3 bucket.") + .required(false) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + public static final ConnectorPropertyDescriptor S3_ENDPOINT_OVERRIDE_URL = new ConnectorPropertyDescriptor.Builder() + .name("S3 Endpoint Override URL") + .description("An optional endpoint URL to use instead of the default AWS S3 endpoint. " + + "This can be used to connect to S3-compatible storage systems but should be left unset for connecting to S3.") + .required(false) + .addValidator(StandardValidators.URL_VALIDATOR) + .build(); + + public static final ConnectorPropertyDescriptor TARGET_OBJECT_SIZE = new ConnectorPropertyDescriptor.Builder() + .name("Target Object Size") + .description("The target size for each object written to S3. The connector will attempt to " + + "combine messages until this size is reached before writing an object to S3. Note that this size is approximate " + + "and may vary from object to object.") + .required(true) + .defaultValue("256 MB") + .addValidator(StandardValidators.DATA_SIZE_VALIDATOR) + .build(); + + public static final ConnectorPropertyDescriptor MERGE_LATENCY = new ConnectorPropertyDescriptor.Builder() + .name("Merge Latency") + .description("The maximum amount of time to wait while merging messages before writing an object to S3. " + + "If this time is reached before the target object size is met, the current set of merged messages " + + "will be written to S3.") + .required(true) + .defaultValue("5 min") + .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR) + .build(); + + public static final ConnectorPropertyDescriptor S3_AUTHENTICATION_STRATEGY = new ConnectorPropertyDescriptor.Builder() + .name("S3 Authentication Strategy") + .description("The authentication strategy to use when connecting to S3.") + .required(true) + .defaultValue(ACCESS_KEY_ID_SECRET_KEY) + .allowableValues(ACCESS_KEY_ID_SECRET_KEY, DEFAULT_CREDENTIALS) + .build(); + + public static final ConnectorPropertyDescriptor S3_ACCESS_KEY_ID = new ConnectorPropertyDescriptor.Builder() + .name("S3 Access Key ID") + .description("The AWS Access Key ID used to authenticate to S3.") + .required(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .dependsOn(S3_AUTHENTICATION_STRATEGY, ACCESS_KEY_ID_SECRET_KEY) + .build(); + + public static final ConnectorPropertyDescriptor S3_SECRET_ACCESS_KEY = new ConnectorPropertyDescriptor.Builder() + .name("S3 Secret Access Key") + .description("The AWS Secret Access Key used to authenticate to S3.") + .required(true) + .type(PropertyType.SECRET) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .dependsOn(S3_AUTHENTICATION_STRATEGY, ACCESS_KEY_ID_SECRET_KEY) + .build(); + + + public static final ConnectorPropertyGroup S3_DESTINATION_GROUP = new ConnectorPropertyGroup.Builder() + .name("S3 Destination Configuration") + .description("Properties required to connect to S3 and specify the target bucket.") + .properties(List.of( + S3_BUCKET, + S3_PREFIX, + S3_REGION, + S3_DATA_FORMAT, + S3_ENDPOINT_OVERRIDE_URL + )) + .build(); + + public static final ConnectorPropertyGroup S3_CREDENTIALS_GROUP = new ConnectorPropertyGroup.Builder() + .name("S3 Credentials") + .description("Properties required to authenticate to S3.") + .properties(List.of( + S3_AUTHENTICATION_STRATEGY, + S3_ACCESS_KEY_ID, + S3_SECRET_ACCESS_KEY + )) + .build(); + + public static final ConnectorPropertyGroup MERGE_GROUP = new ConnectorPropertyGroup.Builder() + .name("Merge Configuration") + .description("Configuration for how data should be merged together before being persisted to S3.") + .properties(List.of( + TARGET_OBJECT_SIZE, + MERGE_LATENCY + )) + .build(); + + public static final ConfigurationStep S3_STEP = new ConfigurationStep.Builder() + .name(S3_STEP_NAME) + .description("Configure connection to S3 and target bucket details.") + .propertyGroups(List.of( + S3_DESTINATION_GROUP, + MERGE_GROUP, + S3_CREDENTIALS_GROUP + )) + .build(); +} diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector new file mode 100644 index 000000000000..d5b0fe6db690 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.connectors.kafkas3.KafkaToS3 + diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/Kafka_Connection.md b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/Kafka_Connection.md new file mode 100644 index 000000000000..a84b962d1962 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/Kafka_Connection.md @@ -0,0 +1,41 @@ + +# Kafka Connection Configuration + +This step configures the connection to your Apache Kafka cluster. + +## Kafka Server Settings + +Enter the bootstrap servers for your Kafka cluster. You can specify multiple brokers +as a comma-separated list (e.g., `broker1:9092,broker2:9092,broker3:9092`). + +### Security Configuration + +Select the appropriate security protocol based on your Kafka cluster configuration: + +| Protocol | Description | +|----------|-------------| +| PLAINTEXT | No encryption or authentication | +| SSL | TLS encryption without SASL authentication | +| SASL_PLAINTEXT | SASL authentication without encryption | +| SASL_SSL | Both SASL authentication and TLS encryption (recommended) | + +If using SASL authentication, provide your username and password credentials. + +## Schema Registry (Optional) + +If your Kafka topics use Avro, Protobuf, or JSON Schema, configure the Schema Registry +URL to enable schema-based serialization and deserialization. + diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/Kafka_Topics.md b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/Kafka_Topics.md new file mode 100644 index 000000000000..aa511e6c4a16 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/Kafka_Topics.md @@ -0,0 +1,48 @@ + +# Kafka Topics Configuration + +This step configures which Kafka topics to consume from and how to consume them. + +## Topic Names + +Select one or more Kafka topics to consume from. The connector will automatically +fetch the list of available topics from your Kafka cluster. + +## Consumer Group ID + +Specify a unique consumer group ID for this connector. Kafka uses consumer groups +to track message offsets and ensure each message is processed only once within a group. + +**Best Practice:** Use a descriptive name that identifies the purpose of this connector, +such as `kafka-to-s3-production` or `analytics-pipeline-consumer`. + +## Offset Reset + +Controls the behavior when no prior offset exists or the current offset is invalid: + +| Value | Description | +|-------|-------------| +| earliest | Start reading from the oldest available message | +| latest | Start reading from the newest messages only | +| none | Fail if no prior offset exists | + +## Kafka Data Format + +Specify the format of messages in your Kafka topics: + +- **Avro**: Messages are encoded using Apache Avro (requires Schema Registry) +- **JSON**: Messages are plain JSON objects + diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/S3_Configuration.md b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/S3_Configuration.md new file mode 100644 index 000000000000..ea3391828c10 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/docs/org.apache.nifi.connectors.kafkas3.KafkaToS3/steps/S3_Configuration.md @@ -0,0 +1,79 @@ + +# S3 Configuration + +This step configures the connection to Amazon S3 or an S3-compatible storage system. + +## S3 Destination Configuration + +### S3 Bucket + +The name of the S3 bucket where data will be written. Ensure the bucket exists +and your credentials have write permissions. + +### S3 Prefix + +An optional prefix (folder path) to prepend to all object keys. For example, +setting this to `kafka-data/` will result in objects like: +`kafka-data/2024/01/15/data-001.json` + +### S3 Region + +The AWS region where your S3 bucket is located (e.g., `us-east-1`, `eu-west-1`). + +### S3 Data Format + +The format to use when writing objects to S3: + +- **JSON**: Write data as JSON objects (human-readable) +- **Avro**: Write data in Apache Avro format (compact, schema-embedded) + +### S3 Endpoint Override URL + +Leave this blank for standard AWS S3. Use this field only when connecting to +S3-compatible storage systems like MinIO, LocalStack, or Ceph. + +## Merge Configuration + +### Target Object Size + +The connector merges multiple Kafka messages together before writing to S3 to +reduce the number of objects created. Specify the target size for merged objects +(e.g., `256 MB`, `1 GB`). + +### Merge Latency + +The maximum time to wait while collecting messages before writing to S3. Even if +the target size hasn't been reached, data will be written after this duration to +ensure timely data availability. + +## S3 Credentials + +### Authentication Strategy + +Choose how to authenticate with AWS: + +| Strategy | Description | +|----------|-------------| +| Access Key ID and Secret Key | Use explicit AWS access credentials | +| Default AWS Credentials | Use the default credential chain (environment variables, IAM roles, etc.) | + +### S3 Access Key ID / Secret Access Key + +When using explicit credentials, provide your AWS access key ID and secret access key. + +**Security Note:** Store sensitive credentials using NiFi's sensitive property +protection or parameter contexts with secret providers. + diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/flows/Kafka_to_S3.json b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/flows/Kafka_to_S3.json new file mode 100644 index 000000000000..fabee83a2abf --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-connector/src/main/resources/flows/Kafka_to_S3.json @@ -0,0 +1,2105 @@ +{ + "flowContents": { + "identifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b", + "instanceIdentifier": "2c825ae7-019a-1000-f3b7-8275175b7b5b", + "name": "Kafka to S3", + "comments": "", + "position": { + "x": -355.0, + "y": -238.5 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "4ce9587d-a171-3dff-bea8-bc67f4b356cd", + "instanceIdentifier": "40e02539-8a55-3a44-53d9-e9d41f7db23a", + "name": "MergeRecord", + "comments": "", + "position": { + "x": -250.0, + "y": -55.5 + }, + "type": "org.apache.nifi.processors.standard.MergeRecord", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "max-bin-age": "#{Merge Latency}", + "correlation-attribute-name": "kafka.topic", + "Attribute Strategy": "Keep Only Common Attributes", + "min-records": "1", + "max-bin-size": "#{Maximum Object size}", + "record-writer": "2384a9ae-311e-3d5f-983b-13f82174244c", + "merge-strategy": "Bin-Packing Algorithm", + "record-reader": "cd697ee5-9621-3680-a6b8-af3237472b5b", + "max-records": "1000000000", + "min-bin-size": "#{Target Object Size}", + "max.bin.count": "10" + }, + "propertyDescriptors": { + "max-bin-age": { + "name": "max-bin-age", + "displayName": "Max Bin Age", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "correlation-attribute-name": { + "name": "correlation-attribute-name", + "displayName": "Correlation Attribute Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Attribute Strategy": { + "name": "Attribute Strategy", + "displayName": "Attribute Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "min-records": { + "name": "min-records", + "displayName": "Minimum Number of Records", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "max-bin-size": { + "name": "max-bin-size", + "displayName": "Maximum Bin Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "record-writer": { + "name": "record-writer", + "displayName": "Record Writer", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "merge-strategy": { + "name": "merge-strategy", + "displayName": "Merge Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "record-reader": { + "name": "record-reader", + "displayName": "Record Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "max-records": { + "name": "max-records", + "displayName": "Maximum Number of Records", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "min-bin-size": { + "name": "min-bin-size", + "displayName": "Minimum Bin Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "max.bin.count": { + "name": "max.bin.count", + "displayName": "Maximum Number of Bins", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "original", + "failure" + ], + "scheduledState": "ENABLED", + "retryCount": 10000, + "retriedRelationships": [ + "failure" + ], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "e427a2b2-355c-3ede-9749-b3e4a8561d22", + "instanceIdentifier": "e048d449-7da8-30f3-8ff4-71c87c1d73db", + "name": "ConsumeKafka", + "comments": "", + "position": { + "x": -250.0, + "y": -268.5 + }, + "type": "org.apache.nifi.kafka.processors.ConsumeKafka", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-kafka-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Topics": "#{Topic Names}", + "Commit Offsets": "true", + "Header Name Pattern": null, + "Key Format": "byte-array", + "Key Record Reader": null, + "Message Demarcator": null, + "Record Reader": "919b2a35-4bc7-380c-b592-6ad2ffffbed5", + "Key Attribute Encoding": "utf-8", + "Max Uncommitted Size": null, + "Topic Format": "names", + "Header Encoding": "UTF-8", + "Max Uncommitted Time": "5 secs", + "Kafka Connection Service": "ccf048dc-004a-35be-8e4d-c7a763a30b02", + "Separate By Key": "false", + "Processing Strategy": "RECORD", + "Record Writer": "2384a9ae-311e-3d5f-983b-13f82174244c", + "Group ID": "#{Consumer Group ID}", + "auto.offset.reset": "#{Kafka Auto Offset Reset}", + "Output Strategy": "USE_VALUE" + }, + "propertyDescriptors": { + "Topics": { + "name": "Topics", + "displayName": "Topics", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Commit Offsets": { + "name": "Commit Offsets", + "displayName": "Commit Offsets", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Header Name Pattern": { + "name": "Header Name Pattern", + "displayName": "Header Name Pattern", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Key Format": { + "name": "Key Format", + "displayName": "Key Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Key Record Reader": { + "name": "Key Record Reader", + "displayName": "Key Record Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Message Demarcator": { + "name": "Message Demarcator", + "displayName": "Message Demarcator", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Record Reader": { + "name": "Record Reader", + "displayName": "Record Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Key Attribute Encoding": { + "name": "Key Attribute Encoding", + "displayName": "Key Attribute Encoding", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Max Uncommitted Size": { + "name": "Max Uncommitted Size", + "displayName": "Max Uncommitted Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Topic Format": { + "name": "Topic Format", + "displayName": "Topic Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Header Encoding": { + "name": "Header Encoding", + "displayName": "Header Encoding", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Max Uncommitted Time": { + "name": "Max Uncommitted Time", + "displayName": "Max Uncommitted Time", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Kafka Connection Service": { + "name": "Kafka Connection Service", + "displayName": "Kafka Connection Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Separate By Key": { + "name": "Separate By Key", + "displayName": "Separate By Key", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Processing Strategy": { + "name": "Processing Strategy", + "displayName": "Processing Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Record Writer": { + "name": "Record Writer", + "displayName": "Record Writer", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Group ID": { + "name": "Group ID", + "displayName": "Group ID", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "auto.offset.reset": { + "name": "auto.offset.reset", + "displayName": "Auto Offset Reset", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Output Strategy": { + "name": "Output Strategy", + "displayName": "Output Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "parse failure" + ], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "501f25bb-cef8-328f-bfc2-fe58ad12e7b6", + "instanceIdentifier": "3de30158-1d3e-38d9-6e68-9578471aa26d", + "name": "PutS3Object", + "comments": "", + "position": { + "x": -250.0, + "y": 169.5 + }, + "type": "org.apache.nifi.processors.aws.s3.PutS3Object", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-aws-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "FullControl User List": "${s3.permissions.full.users}", + "Owner": "${s3.owner}", + "proxy-configuration-service": null, + "Resource Transfer Source": "FLOWFILE_CONTENT", + "Cache Control": null, + "Endpoint Override URL": "#{S3 Endpoint Override URL}", + "File Resource Service": null, + "Multipart Upload Max Age Threshold": "7 days", + "Object Tags Prefix": null, + "Signer Override": "Default Signature", + "Content Type": "${mime.type}", + "Canned ACL": "${s3.permissions.cannedacl}", + "Use Path Style Access": "false", + "Encryption Service": null, + "Write ACL User List": "${s3.permissions.writeacl.users}", + "Use Chunked Encoding": "true", + "Read ACL User List": "${s3.permissions.readacl.users}", + "Content Disposition": null, + "Storage Class": "Standard", + "Multipart Part Size": "5 GB", + "Remove Tag Prefix": "false", + "Object Key": "${#{S3 Prefix}:append(\"/\"):replaceAll('^/$', '')}${filename}", + "AWS Credentials Provider service": "3f5da96c-9c54-39aa-8c84-e8098a44dfab", + "Multipart Threshold": "5 GB", + "Temporary Directory Multipart State": "${java.io.tmpdir}", + "Custom Signer Module Location": null, + "SSL Context Service": null, + "Server Side Encryption": "None", + "Bucket": "#{S3 Bucket}", + "Multipart Upload AgeOff Interval": "60 min", + "Write Permission User List": "${s3.permissions.write.users}", + "Communications Timeout": "30 secs", + "Region": "#{S3 Region}", + "Read Permission User List": "${s3.permissions.read.users}", + "Custom Signer Class Name": null, + "Expiration Time Rule": null + }, + "propertyDescriptors": { + "FullControl User List": { + "name": "FullControl User List", + "displayName": "FullControl User List", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Owner": { + "name": "Owner", + "displayName": "Owner", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "proxy-configuration-service": { + "name": "proxy-configuration-service", + "displayName": "Proxy Configuration Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Resource Transfer Source": { + "name": "Resource Transfer Source", + "displayName": "Resource Transfer Source", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Cache Control": { + "name": "Cache Control", + "displayName": "Cache Control", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Endpoint Override URL": { + "name": "Endpoint Override URL", + "displayName": "Endpoint Override URL", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "File Resource Service": { + "name": "File Resource Service", + "displayName": "File Resource Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Multipart Upload Max Age Threshold": { + "name": "Multipart Upload Max Age Threshold", + "displayName": "Multipart Upload Max Age Threshold", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Object Tags Prefix": { + "name": "Object Tags Prefix", + "displayName": "Object Tags Prefix", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Signer Override": { + "name": "Signer Override", + "displayName": "Signer Override", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Content Type": { + "name": "Content Type", + "displayName": "Content Type", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Canned ACL": { + "name": "Canned ACL", + "displayName": "Canned ACL", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Use Path Style Access": { + "name": "Use Path Style Access", + "displayName": "Use Path Style Access", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Encryption Service": { + "name": "Encryption Service", + "displayName": "Encryption Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Write ACL User List": { + "name": "Write ACL User List", + "displayName": "Write ACL User List", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Use Chunked Encoding": { + "name": "Use Chunked Encoding", + "displayName": "Use Chunked Encoding", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Read ACL User List": { + "name": "Read ACL User List", + "displayName": "Read ACL User List", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Content Disposition": { + "name": "Content Disposition", + "displayName": "Content Disposition", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Storage Class": { + "name": "Storage Class", + "displayName": "Storage Class", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Multipart Part Size": { + "name": "Multipart Part Size", + "displayName": "Multipart Part Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Remove Tag Prefix": { + "name": "Remove Tag Prefix", + "displayName": "Remove Tag Prefix", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Object Key": { + "name": "Object Key", + "displayName": "Object Key", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "AWS Credentials Provider service": { + "name": "AWS Credentials Provider service", + "displayName": "AWS Credentials Provider service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Multipart Threshold": { + "name": "Multipart Threshold", + "displayName": "Multipart Threshold", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Temporary Directory Multipart State": { + "name": "Temporary Directory Multipart State", + "displayName": "Temporary Directory Multipart State", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Custom Signer Module Location": { + "name": "Custom Signer Module Location", + "displayName": "Custom Signer Module Location", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false, + "resourceDefinition": { + "cardinality": "MULTIPLE", + "resourceTypes": [ + "FILE", + "DIRECTORY" + ] + } + }, + "SSL Context Service": { + "name": "SSL Context Service", + "displayName": "SSL Context Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Server Side Encryption": { + "name": "Server Side Encryption", + "displayName": "Server Side Encryption", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Bucket": { + "name": "Bucket", + "displayName": "Bucket", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Multipart Upload AgeOff Interval": { + "name": "Multipart Upload AgeOff Interval", + "displayName": "Multipart Upload AgeOff Interval", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Write Permission User List": { + "name": "Write Permission User List", + "displayName": "Write Permission User List", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Communications Timeout": { + "name": "Communications Timeout", + "displayName": "Communications Timeout", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Region": { + "name": "Region", + "displayName": "Region", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Read Permission User List": { + "name": "Read Permission User List", + "displayName": "Read Permission User List", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Custom Signer Class Name": { + "name": "Custom Signer Class Name", + "displayName": "Custom Signer Class Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Expiration Time Rule": { + "name": "Expiration Time Rule", + "displayName": "Expiration Time Rule", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "success", + "failure" + ], + "scheduledState": "ENABLED", + "retryCount": 100000, + "retriedRelationships": [ + "failure" + ], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + } + ], + "inputPorts": [], + "outputPorts": [], + "connections": [ + { + "identifier": "4bd001fc-7fa6-3c34-b297-bae4987889ad", + "instanceIdentifier": "d6a7b460-571b-39b3-b55f-7d82f62b397a", + "name": "", + "source": { + "id": "e427a2b2-355c-3ede-9749-b3e4a8561d22", + "type": "PROCESSOR", + "groupId": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b", + "name": "ConsumeKafka", + "comments": "", + "instanceIdentifier": "e048d449-7da8-30f3-8ff4-71c87c1d73db" + }, + "destination": { + "id": "4ce9587d-a171-3dff-bea8-bc67f4b356cd", + "type": "PROCESSOR", + "groupId": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b", + "name": "MergeRecord", + "comments": "", + "instanceIdentifier": "40e02539-8a55-3a44-53d9-e9d41f7db23a" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "08803783-a109-3362-9177-0f2016086a2a", + "instanceIdentifier": "45276df0-1076-39a8-7d84-097b7f593486", + "name": "", + "source": { + "id": "4ce9587d-a171-3dff-bea8-bc67f4b356cd", + "type": "PROCESSOR", + "groupId": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b", + "name": "MergeRecord", + "comments": "", + "instanceIdentifier": "40e02539-8a55-3a44-53d9-e9d41f7db23a" + }, + "destination": { + "id": "501f25bb-cef8-328f-bfc2-fe58ad12e7b6", + "type": "PROCESSOR", + "groupId": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b", + "name": "PutS3Object", + "comments": "", + "instanceIdentifier": "3de30158-1d3e-38d9-6e68-9578471aa26d" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "merged" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [ + { + "identifier": "cd697ee5-9621-3680-a6b8-af3237472b5b", + "instanceIdentifier": "2cdfec74-019a-1000-2786-edf80b919dfc", + "name": "Merge JSON Reader", + "comments": "", + "type": "org.apache.nifi.json.JsonTreeReader", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-record-serialization-services-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Schema Branch": null, + "Schema Reference Reader": null, + "Schema Text": "${avro.schema}", + "Max String Length": "20 MB", + "schema-application-strategy": "SELECTED_PART", + "Schema Registry": "34d29160-0744-373c-a1c7-39c548e7e6b1", + "Schema Access Strategy": "schema-name", + "Schema Name": "${schema.name}", + "Timestamp Format": null, + "schema-inference-cache": null, + "Date Format": null, + "starting-field-strategy": "ROOT_NODE", + "Schema Version": null, + "starting-field-name": null, + "Time Format": null, + "Allow Comments": "false" + }, + "propertyDescriptors": { + "Schema Branch": { + "name": "Schema Branch", + "displayName": "Schema Branch", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Reader": { + "name": "Schema Reference Reader", + "displayName": "Schema Reference Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Text": { + "name": "Schema Text", + "displayName": "Schema Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Max String Length": { + "name": "Max String Length", + "displayName": "Max String Length", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "schema-application-strategy": { + "name": "schema-application-strategy", + "displayName": "Schema Application Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Registry": { + "name": "Schema Registry", + "displayName": "Schema Registry", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Access Strategy": { + "name": "Schema Access Strategy", + "displayName": "Schema Access Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Name": { + "name": "Schema Name", + "displayName": "Schema Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Timestamp Format": { + "name": "Timestamp Format", + "displayName": "Timestamp Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "schema-inference-cache": { + "name": "schema-inference-cache", + "displayName": "Schema Inference Cache", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Date Format": { + "name": "Date Format", + "displayName": "Date Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "starting-field-strategy": { + "name": "starting-field-strategy", + "displayName": "Starting Field Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Version": { + "name": "Schema Version", + "displayName": "Schema Version", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "starting-field-name": { + "name": "starting-field-name", + "displayName": "Starting Field Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Time Format": { + "name": "Time Format", + "displayName": "Time Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Allow Comments": { + "name": "Allow Comments", + "displayName": "Allow Comments", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.serialization.RecordReaderFactory", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "b01590c6-5149-3c50-9d16-3a8e7b1bcdf6", + "instanceIdentifier": "2f0a61a3-b66d-3969-09d7-df35dd0b2740", + "name": "AvroRecordSetWriter", + "comments": "", + "type": "org.apache.nifi.avro.AvroRecordSetWriter", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-record-serialization-services-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Schema Branch": null, + "Schema Reference Reader": null, + "Schema Text": "${avro.schema}", + "Schema Registry": null, + "Schema Access Strategy": "inherit-record-schema", + "Schema Name": "${schema.name}", + "compression-format": "NONE", + "Schema Write Strategy": "avro-embedded", + "cache-size": "1000", + "Schema Cache": null, + "Schema Version": null, + "encoder-pool-size": "32", + "Schema Reference Writer": null + }, + "propertyDescriptors": { + "Schema Branch": { + "name": "Schema Branch", + "displayName": "Schema Branch", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Reader": { + "name": "Schema Reference Reader", + "displayName": "Schema Reference Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Text": { + "name": "Schema Text", + "displayName": "Schema Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Registry": { + "name": "Schema Registry", + "displayName": "Schema Registry", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Access Strategy": { + "name": "Schema Access Strategy", + "displayName": "Schema Access Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Name": { + "name": "Schema Name", + "displayName": "Schema Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "compression-format": { + "name": "compression-format", + "displayName": "Compression Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Write Strategy": { + "name": "Schema Write Strategy", + "displayName": "Schema Write Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "cache-size": { + "name": "cache-size", + "displayName": "Cache Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Cache": { + "name": "Schema Cache", + "displayName": "Schema Cache", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Version": { + "name": "Schema Version", + "displayName": "Schema Version", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "encoder-pool-size": { + "name": "encoder-pool-size", + "displayName": "Encoder Pool Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Writer": { + "name": "Schema Reference Writer", + "displayName": "Schema Reference Writer", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.serialization.RecordSetWriterFactory", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "919b2a35-4bc7-380c-b592-6ad2ffffbed5", + "instanceIdentifier": "c169a376-4b21-3062-e8fb-1d243dd0e5df", + "name": "Kafka JSON Reader", + "comments": "", + "type": "org.apache.nifi.json.JsonTreeReader", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-record-serialization-services-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Schema Branch": null, + "Schema Reference Reader": "ecba6fe5-6beb-3b68-bad9-163a3f14c4c8", + "Schema Text": "${avro.schema}", + "Max String Length": "20 MB", + "schema-application-strategy": "SELECTED_PART", + "Schema Registry": "34d29160-0744-373c-a1c7-39c548e7e6b1", + "Schema Access Strategy": "schema-reference-reader", + "Schema Name": "${schema.name}", + "Timestamp Format": null, + "schema-inference-cache": null, + "Date Format": null, + "starting-field-strategy": "ROOT_NODE", + "Schema Version": null, + "starting-field-name": null, + "Time Format": null, + "Allow Comments": "false" + }, + "propertyDescriptors": { + "Schema Branch": { + "name": "Schema Branch", + "displayName": "Schema Branch", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Reader": { + "name": "Schema Reference Reader", + "displayName": "Schema Reference Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Text": { + "name": "Schema Text", + "displayName": "Schema Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Max String Length": { + "name": "Max String Length", + "displayName": "Max String Length", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "schema-application-strategy": { + "name": "schema-application-strategy", + "displayName": "Schema Application Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Registry": { + "name": "Schema Registry", + "displayName": "Schema Registry", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Access Strategy": { + "name": "Schema Access Strategy", + "displayName": "Schema Access Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Name": { + "name": "Schema Name", + "displayName": "Schema Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Timestamp Format": { + "name": "Timestamp Format", + "displayName": "Timestamp Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "schema-inference-cache": { + "name": "schema-inference-cache", + "displayName": "Schema Inference Cache", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Date Format": { + "name": "Date Format", + "displayName": "Date Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "starting-field-strategy": { + "name": "starting-field-strategy", + "displayName": "Starting Field Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Version": { + "name": "Schema Version", + "displayName": "Schema Version", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "starting-field-name": { + "name": "starting-field-name", + "displayName": "Starting Field Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Time Format": { + "name": "Time Format", + "displayName": "Time Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Allow Comments": { + "name": "Allow Comments", + "displayName": "Allow Comments", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.serialization.RecordReaderFactory", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "174d0040-161d-39dc-8da9-0f127fc46373", + "instanceIdentifier": "bf48cf99-1cae-375b-bfe1-bd9aa7368984", + "name": "AvroReader", + "comments": "", + "type": "org.apache.nifi.avro.AvroReader", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-record-serialization-services-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Schema Branch": null, + "Schema Reference Reader": "ecba6fe5-6beb-3b68-bad9-163a3f14c4c8", + "Schema Text": "${avro.schema}", + "cache-size": "1000", + "Schema Version": null, + "Schema Registry": "34d29160-0744-373c-a1c7-39c548e7e6b1", + "Schema Access Strategy": "schema-reference-reader", + "Schema Name": "${schema.name}" + }, + "propertyDescriptors": { + "Schema Branch": { + "name": "Schema Branch", + "displayName": "Schema Branch", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Reader": { + "name": "Schema Reference Reader", + "displayName": "Schema Reference Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Text": { + "name": "Schema Text", + "displayName": "Schema Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "cache-size": { + "name": "cache-size", + "displayName": "Cache Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Version": { + "name": "Schema Version", + "displayName": "Schema Version", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Registry": { + "name": "Schema Registry", + "displayName": "Schema Registry", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Access Strategy": { + "name": "Schema Access Strategy", + "displayName": "Schema Access Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Name": { + "name": "Schema Name", + "displayName": "Schema Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.serialization.RecordReaderFactory", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "3f5da96c-9c54-39aa-8c84-e8098a44dfab", + "instanceIdentifier": "7ea5a25b-89be-3652-9778-e30a69f84806", + "name": "AWSCredentialsProviderControllerService", + "comments": "", + "type": "org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-aws-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Assume Role STS Region": "us-west-2", + "Assume Role Proxy Configuration Service": null, + "Access Key ID": "#{S3 Access Key ID}", + "Assume Role ARN": null, + "Assume Role Session Time": "3600", + "Assume Role Session Name": null, + "Custom Signer Module Location": null, + "Use Default Credentials": "false", + "Assume Role External ID": null, + "Assume Role STS Endpoint Override": null, + "Profile Name": null, + "Credentials File": null, + "Custom Signer Class Name": null, + "Assume Role STS Signer Override": "Default Signature", + "Use Anonymous Credentials": "false", + "Secret Access Key": "#{S3 Secret Access Key}", + "Assume Role SSL Context Service": null + }, + "propertyDescriptors": { + "Assume Role STS Region": { + "name": "Assume Role STS Region", + "displayName": "Assume Role STS Region", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Assume Role Proxy Configuration Service": { + "name": "Assume Role Proxy Configuration Service", + "displayName": "Assume Role Proxy Configuration Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Access Key ID": { + "name": "Access Key ID", + "displayName": "Access Key ID", + "identifiesControllerService": false, + "sensitive": true, + "dynamic": false + }, + "Assume Role ARN": { + "name": "Assume Role ARN", + "displayName": "Assume Role ARN", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Assume Role Session Time": { + "name": "Assume Role Session Time", + "displayName": "Assume Role Session Time", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Assume Role Session Name": { + "name": "Assume Role Session Name", + "displayName": "Assume Role Session Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Custom Signer Module Location": { + "name": "Custom Signer Module Location", + "displayName": "Custom Signer Module Location", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false, + "resourceDefinition": { + "cardinality": "MULTIPLE", + "resourceTypes": [ + "FILE", + "DIRECTORY" + ] + } + }, + "Use Default Credentials": { + "name": "Use Default Credentials", + "displayName": "Use Default Credentials", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Assume Role External ID": { + "name": "Assume Role External ID", + "displayName": "Assume Role External ID", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Assume Role STS Endpoint Override": { + "name": "Assume Role STS Endpoint Override", + "displayName": "Assume Role STS Endpoint Override", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Profile Name": { + "name": "Profile Name", + "displayName": "Profile Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Credentials File": { + "name": "Credentials File", + "displayName": "Credentials File", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false, + "resourceDefinition": { + "cardinality": "SINGLE", + "resourceTypes": [ + "FILE" + ] + } + }, + "Custom Signer Class Name": { + "name": "Custom Signer Class Name", + "displayName": "Custom Signer Class Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Assume Role STS Signer Override": { + "name": "Assume Role STS Signer Override", + "displayName": "Assume Role STS Signer Override", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Use Anonymous Credentials": { + "name": "Use Anonymous Credentials", + "displayName": "Use Anonymous Credentials", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Secret Access Key": { + "name": "Secret Access Key", + "displayName": "Secret Access Key", + "identifiesControllerService": false, + "sensitive": true, + "dynamic": false + }, + "Assume Role SSL Context Service": { + "name": "Assume Role SSL Context Service", + "displayName": "Assume Role SSL Context Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-aws-service-api-nar", + "version": "2.9.0-SNAPSHOT" + } + }, + { + "type": "org.apache.nifi.processors.aws.credentials.provider.AwsCredentialsProviderService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-aws-service-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "2384a9ae-311e-3d5f-983b-13f82174244c", + "instanceIdentifier": "d02eb508-0c95-31df-b86b-e1adadd77845", + "name": "JsonRecordSetWriter", + "comments": "", + "type": "org.apache.nifi.json.JsonRecordSetWriter", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-record-serialization-services-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "Schema Branch": null, + "Schema Reference Reader": null, + "Schema Text": "${avro.schema}", + "Allow Scientific Notation": "false", + "compression-level": "1", + "Schema Registry": null, + "Schema Access Strategy": "inherit-record-schema", + "Schema Name": "${schema.name}", + "Timestamp Format": null, + "Date Format": null, + "Pretty Print JSON": "false", + "compression-format": "none", + "Schema Write Strategy": "schema-name", + "suppress-nulls": "never-suppress", + "output-grouping": "output-oneline", + "Schema Cache": null, + "Schema Version": null, + "Time Format": null, + "Schema Reference Writer": null + }, + "propertyDescriptors": { + "Schema Branch": { + "name": "Schema Branch", + "displayName": "Schema Branch", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Reader": { + "name": "Schema Reference Reader", + "displayName": "Schema Reference Reader", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Text": { + "name": "Schema Text", + "displayName": "Schema Text", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Allow Scientific Notation": { + "name": "Allow Scientific Notation", + "displayName": "Allow Scientific Notation", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "compression-level": { + "name": "compression-level", + "displayName": "Compression Level", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Registry": { + "name": "Schema Registry", + "displayName": "Schema Registry", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Access Strategy": { + "name": "Schema Access Strategy", + "displayName": "Schema Access Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Name": { + "name": "Schema Name", + "displayName": "Schema Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Timestamp Format": { + "name": "Timestamp Format", + "displayName": "Timestamp Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Date Format": { + "name": "Date Format", + "displayName": "Date Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Pretty Print JSON": { + "name": "Pretty Print JSON", + "displayName": "Pretty Print JSON", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "compression-format": { + "name": "compression-format", + "displayName": "Compression Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Write Strategy": { + "name": "Schema Write Strategy", + "displayName": "Schema Write Strategy", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "suppress-nulls": { + "name": "suppress-nulls", + "displayName": "Suppress Null Values", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "output-grouping": { + "name": "output-grouping", + "displayName": "Output Grouping", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Cache": { + "name": "Schema Cache", + "displayName": "Schema Cache", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Schema Version": { + "name": "Schema Version", + "displayName": "Schema Version", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Time Format": { + "name": "Time Format", + "displayName": "Time Format", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Reference Writer": { + "name": "Schema Reference Writer", + "displayName": "Schema Reference Writer", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.serialization.RecordSetWriterFactory", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "34d29160-0744-373c-a1c7-39c548e7e6b1", + "instanceIdentifier": "e0f7c153-e070-369b-451d-745b53ebe26a", + "name": "ConfluentSchemaRegistry", + "comments": "", + "type": "org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-confluent-platform-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "SSL Context Service": null, + "Username": "#{Schema Registry Username}", + "Schema Registry URLs": "#{Schema Registry URLs}", + "Communications Timeout": "30 secs", + "Cache Size": "1000", + "Authentication Type": "#{Schema Registry Authentication Type}", + "Cache Expiration": "1 hour", + "Password": "#{Schema Registry Password}" + }, + "propertyDescriptors": { + "SSL Context Service": { + "name": "SSL Context Service", + "displayName": "SSL Context Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "Username": { + "name": "Username", + "displayName": "Username", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Schema Registry URLs": { + "name": "Schema Registry URLs", + "displayName": "Schema Registry URLs", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Communications Timeout": { + "name": "Communications Timeout", + "displayName": "Communications Timeout", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Cache Size": { + "name": "Cache Size", + "displayName": "Cache Size", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Authentication Type": { + "name": "Authentication Type", + "displayName": "Authentication Type", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Cache Expiration": { + "name": "Cache Expiration", + "displayName": "Cache Expiration", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "Password": { + "name": "Password", + "displayName": "Password", + "identifiesControllerService": false, + "sensitive": true, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.schemaregistry.services.SchemaRegistry", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "ecba6fe5-6beb-3b68-bad9-163a3f14c4c8", + "instanceIdentifier": "da19419b-4821-315c-74b2-03d97cf411dc", + "name": "ConfluentEncodedSchemaReferenceReader", + "comments": "", + "type": "org.apache.nifi.confluent.schemaregistry.ConfluentEncodedSchemaReferenceReader", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-confluent-platform-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": {}, + "propertyDescriptors": {}, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.schemaregistry.services.SchemaReferenceReader", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-standard-services-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + }, + { + "identifier": "ccf048dc-004a-35be-8e4d-c7a763a30b02", + "instanceIdentifier": "5549d8f6-48eb-3677-821e-292b9b7e73d2", + "name": "Kafka3ConnectionService", + "comments": "", + "type": "org.apache.nifi.kafka.service.Kafka3ConnectionService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-kafka-3-service-nar", + "version": "2.9.0-SNAPSHOT" + }, + "properties": { + "ack.wait.time": "10 sec", + "sasl.username": "#{Kafka SASL Username}", + "max.poll.records": "20000", + "bootstrap.servers": "#{Kafka Bootstrap Servers}", + "sasl.kerberos.service.name": null, + "security.protocol": "#{Kafka Security Protocol}", + "SSL Context Service": null, + "sasl.mechanism": "#{Kafka SASL Mechanism}", + "isolation.level": "read_committed", + "oauth2-access-token-provider-service": null, + "max.block.ms": "10 sec", + "kerberos-user-service": null, + "sasl.password": "#{Kafka SASL Password}", + "default.api.timeout.ms": "60 sec" + }, + "propertyDescriptors": { + "ack.wait.time": { + "name": "ack.wait.time", + "displayName": "Acknowledgment Wait Time", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "sasl.username": { + "name": "sasl.username", + "displayName": "SASL Username", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "max.poll.records": { + "name": "max.poll.records", + "displayName": "Max Poll Records", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "bootstrap.servers": { + "name": "bootstrap.servers", + "displayName": "Bootstrap Servers", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "sasl.kerberos.service.name": { + "name": "sasl.kerberos.service.name", + "displayName": "Kerberos Service Name", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "security.protocol": { + "name": "security.protocol", + "displayName": "Security Protocol", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "SSL Context Service": { + "name": "SSL Context Service", + "displayName": "SSL Context Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "sasl.mechanism": { + "name": "sasl.mechanism", + "displayName": "SASL Mechanism", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "isolation.level": { + "name": "isolation.level", + "displayName": "Transaction Isolation Level", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "oauth2-access-token-provider-service": { + "name": "oauth2-access-token-provider-service", + "displayName": "OAuth2 Access Token Provider Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "max.block.ms": { + "name": "max.block.ms", + "displayName": "Max Metadata Wait Time", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + }, + "kerberos-user-service": { + "name": "kerberos-user-service", + "displayName": "Kerberos User Service", + "identifiesControllerService": true, + "sensitive": false, + "dynamic": false + }, + "sasl.password": { + "name": "sasl.password", + "displayName": "SASL Password", + "identifiesControllerService": false, + "sensitive": true, + "dynamic": false + }, + "default.api.timeout.ms": { + "name": "default.api.timeout.ms", + "displayName": "Client Timeout", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "controllerServiceApis": [ + { + "type": "org.apache.nifi.kafka.service.api.KafkaConnectionService", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-kafka-service-api-nar", + "version": "2.9.0-SNAPSHOT" + } + } + ], + "scheduledState": "DISABLED", + "bulletinLevel": "WARN", + "componentType": "CONTROLLER_SERVICE", + "groupIdentifier": "b3a5c0de-2130-3ba9-942b-9bae5735bd4b" + } + ], + "parameterContextName": "Kafka to S3", + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "componentType": "PROCESS_GROUP", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE" + }, + "externalControllerServices": {}, + "parameterContexts": { + "Kafka to S3": { + "name": "Kafka to S3", + "parameters": [ + { + "name": "S3 Secret Access Key", + "description": "", + "sensitive": true, + "provided": false + }, + { + "name": "Schema Registry Password", + "description": "", + "sensitive": true, + "provided": false + }, + { + "name": "Consumer Group ID", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "Kafka SASL Mechanism", + "description": "", + "sensitive": false, + "provided": false, + "value": "SCRAM-SHA-512" + }, + { + "name": "Kafka Data Format", + "description": "", + "sensitive": false, + "provided": false, + "value": "JSON" + }, + { + "name": "Schema Registry Username", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "S3 Data Format", + "description": "", + "sensitive": false, + "provided": false, + "value": "JSON" + }, + { + "name": "Schema Registry URLs", + "description": "", + "sensitive": false, + "provided": false, + "value": "http://localhost:8081" + }, + { + "name": "Merge Latency", + "description": "", + "sensitive": false, + "provided": false, + "value": "10 mins" + }, + { + "name": "S3 Endpoint Override URL", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "Schema Registry Authentication Type", + "description": "", + "sensitive": false, + "provided": false, + "value": "BASIC" + }, + { + "name": "Kafka Security Protocol", + "description": "", + "sensitive": false, + "provided": false, + "value": "SASL_SSL" + }, + { + "name": "Maximum Object size", + "description": "", + "sensitive": false, + "provided": false, + "value": "300 MB" + }, + { + "name": "S3 Region", + "description": "", + "sensitive": false, + "provided": false, + "value": "us-west-2" + }, + { + "name": "Kafka SASL Password", + "description": "", + "sensitive": true, + "provided": false + }, + { + "name": "Kafka Bootstrap Servers", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "Kafka Auto Offset Reset", + "description": "", + "sensitive": false, + "provided": false, + "value": "latest" + }, + { + "name": "S3 Bucket", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "Kafka SASL Username", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "S3 Access Key ID", + "description": "", + "sensitive": true, + "provided": false + }, + { + "name": "Target Object Size", + "description": "", + "sensitive": false, + "provided": false, + "value": "256 MB" + }, + { + "name": "Topic Names", + "description": "", + "sensitive": false, + "provided": false + }, + { + "name": "S3 Prefix", + "description": "", + "sensitive": false, + "provided": false + } + ], + "inheritedParameterContexts": [], + "description": "", + "componentType": "PARAMETER_CONTEXT" + } + }, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} \ No newline at end of file diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-integration-tests/pom.xml b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-integration-tests/pom.xml new file mode 100644 index 000000000000..70f12a0652c1 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-integration-tests/pom.xml @@ -0,0 +1,238 @@ + + + + 4.0.0 + + org.apache.nifi + nifi-kafka-to-s3-bundle + 2.9.0-SNAPSHOT + + + nifi-kafka-to-s3-integration-tests + + + + confluent + https://packages.confluent.io/maven/ + + + + + + org.slf4j + slf4j-api + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-connector-mock-api + 2.9.0-SNAPSHOT + + + org.apache.nifi + nifi-connector-mock + 2.9.0-SNAPSHOT + + + + + org.testcontainers + testcontainers-kafka + 2.0.1 + test + + + org.testcontainers + localstack + 1.20.4 + test + + + + + + org.apache.kafka + kafka-clients + 4.1.1 + + + + org.lz4 + lz4-java + + + + + at.yawk.lz4 + lz4-java + 1.10.1 + + + + + io.confluent + kafka-avro-serializer + 7.8.0 + test + + + io.confluent + kafka-schema-registry-client + 7.8.0 + test + + + org.apache.avro + avro + 1.11.3 + test + + + + + software.amazon.awssdk + s3 + 2.36.0 + test + + + software.amazon.awssdk + url-connection-client + 2.36.0 + test + + + + + org.apache.nifi + nifi-kafka-to-s3-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-kafka-3-service-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-kafka-service-api-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-kafka-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-aws-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-aws-service-api-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-standard-services-api-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-standard-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-standard-shared-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-record-serialization-services-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-confluent-platform-nar + 2.9.0-SNAPSHOT + nar + + + + + org.apache.nifi + nifi-framework-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-jetty-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-provenance-repository-nar + 2.9.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-connector-mock-server-nar + 2.9.0-SNAPSHOT + nar + + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + copy-nar + package + + copy-dependencies + + + ${project.build.directory}/libDir + nar + + + + + + + diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-integration-tests/src/test/java/org/apache/nifi/connectors/kafkas3/KafkaToS3IT.java b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-integration-tests/src/test/java/org/apache/nifi/connectors/kafkas3/KafkaToS3IT.java new file mode 100644 index 000000000000..f715ea2e4e7c --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-integration-tests/src/test/java/org/apache/nifi/connectors/kafkas3/KafkaToS3IT.java @@ -0,0 +1,758 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.kafkas3; + +import io.confluent.kafka.serializers.KafkaAvroSerializer; +import org.apache.avro.Schema; +import org.apache.avro.generic.GenericData; +import org.apache.avro.generic.GenericRecord; +import org.apache.kafka.clients.CommonClientConfigs; +import org.apache.kafka.clients.admin.AdminClient; +import org.apache.kafka.clients.admin.AdminClientConfig; +import org.apache.kafka.clients.admin.NewTopic; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.config.SaslConfigs; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.mock.connector.StandardConnectorTestRunner; +import org.apache.nifi.mock.connector.server.ConnectorConfigVerificationResult; +import org.apache.nifi.mock.connector.server.ConnectorTestRunner; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.Network; +import org.testcontainers.containers.localstack.LocalStackContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.kafka.ConfluentKafkaContainer; +import org.testcontainers.utility.DockerImageName; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.ResponseInputStream; +import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.CreateBucketRequest; +import software.amazon.awssdk.services.s3.model.GetObjectRequest; +import software.amazon.awssdk.services.s3.model.GetObjectResponse; +import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; +import software.amazon.awssdk.services.s3.model.ListObjectsV2Response; +import software.amazon.awssdk.services.s3.model.S3Object; + +import java.io.File; +import java.io.IOException; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.concurrent.ExecutionException; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class KafkaToS3IT { + + private static ConnectorTestRunner runner; + private static Network network; + private static ConfluentKafkaContainer kafkaContainer; + private static GenericContainer schemaRegistryContainer; + private static LocalStackContainer localStackContainer; + private static S3Client s3Client; + + private static final String SCRAM_USERNAME = "testuser"; + private static final String SCRAM_PASSWORD = "testpassword"; + + private static final String S3_REGION = "us-west-2"; + + // JAAS configuration for Kafka broker SASL/PLAIN authentication. + // The 'username' and 'password' fields are credentials the broker uses for inter-broker communication. + // The 'user_=""' entries define client users that can authenticate to this broker. + // In this setup: + // - Broker uses 'admin' / 'admin-secret' for inter-broker communication (though we use PLAINTEXT for that) + // - Clients can authenticate using 'testuser' / 'testpassword' on the SASL listener with PLAIN mechanism + private static final String JAAS_CONFIG_CONTENT = """ + KafkaServer { + org.apache.kafka.common.security.plain.PlainLoginModule required + username="admin" + password="admin-secret" + user_%s="%s"; + }; + """.formatted(SCRAM_USERNAME, SCRAM_PASSWORD); + + + @BeforeAll + public static void setupTestContainers() { + network = Network.newNetwork(); + + kafkaContainer = new ConfluentKafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.8.0")) + .withNetwork(network) + .withNetworkAliases("kafka") + .withStartupTimeout(Duration.ofSeconds(10)) + .withEnv("KAFKA_LISTENER_SECURITY_PROTOCOL_MAP", "CONTROLLER:PLAINTEXT,BROKER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SASL:SASL_PLAINTEXT") + .withEnv("KAFKA_LISTENERS", "CONTROLLER://0.0.0.0:9094,BROKER://0.0.0.0:9092,PLAINTEXT://0.0.0.0:19092,SASL://0.0.0.0:9093") + .withEnv("KAFKA_ADVERTISED_LISTENERS", "BROKER://kafka:9092,PLAINTEXT://kafka:19092,SASL://localhost:9093") + .withEnv("KAFKA_CONTROLLER_LISTENER_NAMES", "CONTROLLER") + .withEnv("KAFKA_INTER_BROKER_LISTENER_NAME", "BROKER") + .withEnv("KAFKA_SASL_ENABLED_MECHANISMS", "PLAIN") + .withEnv("KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS", "0") + .withEnv("KAFKA_GROUP_MIN_SESSION_TIMEOUT_MS", "1000") + .withEnv("KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS", "60000") + .withEnv("KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR", "1") + .withEnv("KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR", "1") + .withEnv("KAFKA_TRANSACTION_STATE_LOG_MIN_ISR", "1") + .withEnv("KAFKA_OPTS", "-Djava.security.auth.login.config=/tmp/kafka_jaas.conf") + .withCommand( + "sh", "-c", + "echo '" + JAAS_CONFIG_CONTENT + "' > /tmp/kafka_jaas.conf && " + + "/etc/confluent/docker/run" + ); + + kafkaContainer.setPortBindings(List.of("9093:9093")); + kafkaContainer.start(); + + schemaRegistryContainer = new GenericContainer<>(DockerImageName.parse("confluentinc/cp-schema-registry:7.8.0")) + .withNetwork(network) + .withExposedPorts(8081) + .withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry") + .withEnv("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", "PLAINTEXT://kafka:19092") + .withEnv("SCHEMA_REGISTRY_LISTENERS", "http://0.0.0.0:8081") + .withStartupTimeout(Duration.ofSeconds(60)) + .waitingFor(Wait.forHttp("/subjects").forStatusCode(200)) + .dependsOn(kafkaContainer); + + schemaRegistryContainer.start(); + + localStackContainer = new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.1.0")) + .withServices(LocalStackContainer.Service.S3) + .withStartupTimeout(Duration.ofSeconds(30)); + + localStackContainer.start(); + + s3Client = S3Client.builder() + .endpointOverride(localStackContainer.getEndpoint()) + .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(localStackContainer.getAccessKey(), localStackContainer.getSecretKey()))) + .region(Region.of(S3_REGION)) + .httpClient(UrlConnectionHttpClient.builder().build()) + .forcePathStyle(true) + .build(); + } + + @BeforeEach + public void setupRunner() { + runner = new StandardConnectorTestRunner.Builder() + .connectorClassName("org.apache.nifi.connectors.kafkas3.KafkaToS3") + .narLibraryDirectory(new File("target/libDir")) + .build(); + assertNotNull(runner); + } + + @AfterAll + public static void cleanupTestContainers() { + if (s3Client != null) { + s3Client.close(); + } + + if (localStackContainer != null) { + localStackContainer.stop(); + } + + if (schemaRegistryContainer != null) { + schemaRegistryContainer.stop(); + } + + if (kafkaContainer != null) { + kafkaContainer.stop(); + } + + if (network != null) { + network.close(); + } + } + + @AfterEach + public void cleanupRunner() throws IOException { + if (runner != null) { + runner.close(); + } + } + + + private void createS3Bucket(final String bucketName) { + s3Client.createBucket(CreateBucketRequest.builder().bucket(bucketName).build()); + } + + private void createKafkaTopics(final String... topicNames) throws ExecutionException, InterruptedException { + final Properties adminProps = new Properties(); + adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9093"); + adminProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT"); + adminProps.put(SaslConfigs.SASL_MECHANISM, "PLAIN"); + adminProps.put(SaslConfigs.SASL_JAAS_CONFIG, String.format( + "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";", + SCRAM_USERNAME, SCRAM_PASSWORD + )); + + try (final AdminClient adminClient = AdminClient.create(adminProps)) { + final List topics = new ArrayList<>(); + for (final String topicName : topicNames) { + topics.add(new NewTopic(topicName, 1, (short) 1)); + } + + adminClient.createTopics(topics).all().get(); + } + } + + private void produceRecordsToTopic(final String topicName, final String... records) throws ExecutionException, InterruptedException { + final Properties producerProps = new Properties(); + producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9093"); + producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); + producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); + producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT"); + producerProps.put(SaslConfigs.SASL_MECHANISM, "PLAIN"); + producerProps.put(SaslConfigs.SASL_JAAS_CONFIG, String.format( + "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";", + SCRAM_USERNAME, SCRAM_PASSWORD + )); + + try (final KafkaProducer producer = new KafkaProducer<>(producerProps)) { + for (final String record : records) { + final ProducerRecord producerRecord = new ProducerRecord<>(topicName, record); + producer.send(producerRecord).get(); + } + + producer.flush(); + } + } + + private String getSchemaRegistryUrl() { + return String.format("http://%s:%d", schemaRegistryContainer.getHost(), schemaRegistryContainer.getMappedPort(8081)); + } + + private void produceAvroRecordsToTopic(final String topicName, final Schema schema, final GenericRecord... records) throws ExecutionException, InterruptedException { + final Properties producerProps = new Properties(); + producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9093"); + producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); + producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName()); + producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT"); + producerProps.put(SaslConfigs.SASL_MECHANISM, "PLAIN"); + producerProps.put(SaslConfigs.SASL_JAAS_CONFIG, String.format( + "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";", + SCRAM_USERNAME, SCRAM_PASSWORD + )); + producerProps.put("schema.registry.url", getSchemaRegistryUrl()); + + try (final KafkaProducer producer = new KafkaProducer<>(producerProps)) { + for (final GenericRecord record : records) { + final ProducerRecord producerRecord = new ProducerRecord<>(topicName, record); + producer.send(producerRecord).get(); + } + + producer.flush(); + } + } + + + @Test + public void testVerification() throws ExecutionException, InterruptedException, FlowUpdateException { + createKafkaTopics("topic-1", "topic-2", "topic-3", "topic-4", "topic-5", "Z-topic", "an-important-topic"); + + produceRecordsToTopic("topic-1", + """ + {"id": 1, "name": "Alice", "age": 30}""", + """ + {"id": 2, "name": "Bob", "age": 25}""", + """ + {"id": 3, "name": "Charlie", "age": 35}""" + ); + + produceRecordsToTopic("an-important-topic", + "This is a plaintext message", + "Another important message", + "Final plaintext record" + ); + + final Map kafkaServerConfig = Map.of( + "Kafka Brokers", "localhost:9093", + "Security Protocol", "SASL_PLAINTEXT", + "SASL Mechanism", "PLAIN", + "Username", SCRAM_USERNAME + ); + runner.addSecret("kafka-password", SCRAM_PASSWORD); + final Map kafkaServerSecretRefs = Map.of( + "Password", runner.createSecretReference("kafka-password") + ); + + // Perform verification to ensure that valid server configuration passes + final ConnectorConfigVerificationResult connectionVerificationResults = runner.verifyConfiguration("Kafka Connection", kafkaServerConfig, kafkaServerSecretRefs); + connectionVerificationResults.assertNoFailures(); + + // Apply the configuration that we've now validated + runner.configure("Kafka Connection", kafkaServerConfig, kafkaServerSecretRefs); + + // Perform verification to ensure that valid topic configuration passes + final Map topic1Config = Map.of( + "Topic Names", "topic-1", + "Consumer Group ID", "nifi-kafka-to-s3-testSuccessfulFlow", + "Offset Reset", "earliest", + "Kafka Data Format", "JSON" + ); + final ConnectorConfigVerificationResult topic1VerificationResults = runner.verifyConfiguration("Kafka Topics", topic1Config); + topic1VerificationResults.assertNoFailures(); + + // Perform verification against a topic with invalid data for the selected data format + final Map importantTopicConfig = Map.of( + "Topic Names", "an-important-topic", + "Consumer Group ID", "nifi-kafka-to-s3-testSuccessfulFlow", + "Offset Reset", "earliest", + "Kafka Data Format", "JSON" + ); + + final ConnectorConfigVerificationResult importantTopicVerificationResults = runner.verifyConfiguration("Kafka Topics", importantTopicConfig); + final List invalidImportantTopicResults = importantTopicVerificationResults.getFailedResults(); + assertEquals(1, invalidImportantTopicResults.size()); + final ConfigVerificationResult invalidResult = invalidImportantTopicResults.getFirst(); + assertTrue(invalidResult.getExplanation().contains("parse"), "Unexpected validation reason: " + invalidResult.getExplanation()); + + runner.applyUpdate(); + } + + @Test + public void testJsonFlow() throws IOException, ExecutionException, InterruptedException, FlowUpdateException { + final String bucketName = "test-json-flow"; + createS3Bucket(bucketName); + createKafkaTopics("story"); + + produceRecordsToTopic("story", + """ + {"page": 1, "words": "Once upon a time, there was a NiFi developer." }""", + """ + {"page": 2, "words": "The developer wanted to build a connector to move data from Kafka to S3." }""", + """ + {"page": 3, "words": "After much effort, the connector was complete and worked flawlessly!" }""", + """ + {"page": 4, "words": "The end." }""" + ); + + final Map kafkaServerConfig = Map.of( + "Kafka Brokers", "localhost:9093", + "Security Protocol", "SASL_PLAINTEXT", + "SASL Mechanism", "PLAIN", + "Username", SCRAM_USERNAME + ); + runner.addSecret("kafka-password", SCRAM_PASSWORD); + final Map kafkaSecretRefs = Map.of( + "Password", runner.createSecretReference("kafka-password") + ); + + final Map kafkaTopicConfig = Map.of( + "Topic Names", "story", + "Consumer Group ID", "nifi-kafka-to-s3-testSuccessfulFlow", + "Offset Reset", "earliest", + "Kafka Data Format", "JSON" + ); + + final Map s3Config = Map.ofEntries( + Map.entry("S3 Region", S3_REGION), + Map.entry("S3 Data Format", "Avro"), + Map.entry("S3 Bucket", bucketName), + Map.entry("S3 Endpoint Override URL", localStackContainer.getEndpoint().toString()), + Map.entry("S3 Authentication Strategy", "Access Key ID and Secret Key"), + Map.entry("S3 Access Key ID", localStackContainer.getAccessKey()), + Map.entry("Target Object Size", "1 MB"), + Map.entry("Merge Latency", "1 sec") + ); + runner.addSecret("s3-secret-key", localStackContainer.getSecretKey()); + final Map s3SecretRefs = Map.of( + "S3 Secret Access Key", runner.createSecretReference("s3-secret-key") + ); + + runner.configure("Kafka Connection", kafkaServerConfig, kafkaSecretRefs); + runner.configure("Kafka Topics", kafkaTopicConfig); + runner.configure("S3 Configuration", s3Config, s3SecretRefs); + runner.applyUpdate(); + + final List validationResults = runner.validate(); + assertEquals(Collections.emptyList(), validationResults); + + runner.startConnector(); + try { + runner.waitForDataIngested(Duration.ofSeconds(10)); + runner.waitForIdle(Duration.ofSeconds(30)); + } finally { + runner.stopConnector(); + } + + verifyS3ObjectsCreated(bucketName); + } + + private void verifyS3ObjectsCreated(final String bucketName) throws IOException { + final ListObjectsV2Response listResponse = s3Client.listObjectsV2(ListObjectsV2Request.builder().bucket(bucketName).build()); + final List objects = listResponse.contents(); + + assertFalse(objects.isEmpty(), "Expected at least one object in S3 bucket"); + + for (final S3Object s3Object : objects) { + final GetObjectRequest getObjectRequest = GetObjectRequest.builder().bucket(bucketName).key(s3Object.key()).build(); + try (final ResponseInputStream objectContent = s3Client.getObject(getObjectRequest)) { + final long objectSize = objectContent.response().contentLength(); + assertTrue(objectSize > 0, "Expected S3 object " + s3Object.key() + " to have content"); + } + } + } + + @Test + public void testSchemaRegistryVerification() throws ExecutionException, InterruptedException, FlowUpdateException { + createKafkaTopics("avro-topic"); + + final String schemaString = """ + { + "type": "record", + "name": "TestRecord", + "namespace": "org.apache.nifi.test", + "fields": [ + {"name": "id", "type": "int"}, + {"name": "message", "type": "string"} + ] + }"""; + + final Schema schema = new Schema.Parser().parse(schemaString); + + final GenericRecord record1 = new GenericData.Record(schema); + record1.put("id", 100); + record1.put("message", "Test message 1"); + + final GenericRecord record2 = new GenericData.Record(schema); + record2.put("id", 200); + record2.put("message", "Test message 2"); + + produceAvroRecordsToTopic("avro-topic", schema, record1, record2); + + final Map kafkaConnectionConfig = Map.of( + "Kafka Brokers", "localhost:9093", + "Security Protocol", "SASL_PLAINTEXT", + "SASL Mechanism", "PLAIN", + "Username", SCRAM_USERNAME, + "Schema Registry URL", getSchemaRegistryUrl() + ); + runner.addSecret("kafka-password", SCRAM_PASSWORD); + final Map kafkaSecretRefs = Map.of( + "Password", runner.createSecretReference("kafka-password") + ); + + final ConnectorConfigVerificationResult connectionVerificationResults = runner.verifyConfiguration("Kafka Connection", kafkaConnectionConfig, kafkaSecretRefs); + connectionVerificationResults.assertNoFailures(); + + runner.configure("Kafka Connection", kafkaConnectionConfig, kafkaSecretRefs); + + final Map avroTopicConfig = Map.of( + "Topic Names", "avro-topic", + "Consumer Group ID", "nifi-kafka-to-s3-testSchemaRegistryVerification", + "Offset Reset", "earliest", + "Kafka Data Format", "Avro" + ); + + final ConnectorConfigVerificationResult avroTopicVerificationResults = runner.verifyConfiguration("Kafka Topics", avroTopicConfig); + avroTopicVerificationResults.assertNoFailures(); + + runner.applyUpdate(); + } + + @Test + public void testWithSchemaRegistry() throws IOException, ExecutionException, InterruptedException, FlowUpdateException { + final String bucketName = "test-schema-registry"; + createS3Bucket(bucketName); + createKafkaTopics("user-events"); + + final String schemaString = """ + { + "type": "record", + "name": "UserEvent", + "namespace": "org.apache.nifi.test", + "fields": [ + {"name": "userId", "type": "int"}, + {"name": "userName", "type": "string"}, + {"name": "eventType", "type": "string"}, + {"name": "timestamp", "type": "long"} + ] + }"""; + + final Schema schema = new Schema.Parser().parse(schemaString); + + final GenericRecord record1 = new GenericData.Record(schema); + record1.put("userId", 1001); + record1.put("userName", "alice"); + record1.put("eventType", "login"); + record1.put("timestamp", System.currentTimeMillis()); + + final GenericRecord record2 = new GenericData.Record(schema); + record2.put("userId", 1002); + record2.put("userName", "bob"); + record2.put("eventType", "purchase"); + record2.put("timestamp", System.currentTimeMillis()); + + final GenericRecord record3 = new GenericData.Record(schema); + record3.put("userId", 1003); + record3.put("userName", "charlie"); + record3.put("eventType", "logout"); + record3.put("timestamp", System.currentTimeMillis()); + + produceAvroRecordsToTopic("user-events", schema, record1, record2, record3); + + final Map kafkaConnectionConfig = Map.of( + "Kafka Brokers", "localhost:9093", + "Security Protocol", "SASL_PLAINTEXT", + "SASL Mechanism", "PLAIN", + "Username", SCRAM_USERNAME, + "Schema Registry URL", getSchemaRegistryUrl() + ); + runner.addSecret("kafka-password", SCRAM_PASSWORD); + final Map kafkaSecretRefs = Map.of( + "Password", runner.createSecretReference("kafka-password") + ); + + final Map kafkaTopicConfig = Map.of( + "Topic Names", "user-events", + "Consumer Group ID", "nifi-kafka-to-s3-testSchemaRegistry", + "Offset Reset", "earliest", + "Kafka Data Format", "Avro" + ); + + final Map s3Config = Map.ofEntries( + Map.entry("S3 Region", S3_REGION), + Map.entry("S3 Data Format", "Avro"), + Map.entry("S3 Bucket", bucketName), + Map.entry("S3 Endpoint Override URL", localStackContainer.getEndpoint().toString()), + Map.entry("S3 Authentication Strategy", "Access Key ID and Secret Key"), + Map.entry("S3 Access Key ID", localStackContainer.getAccessKey()), + Map.entry("Target Object Size", "1 MB"), + Map.entry("Merge Latency", "1 sec") + ); + runner.addSecret("s3-secret-key", localStackContainer.getSecretKey()); + final Map s3SecretRefs = Map.of( + "S3 Secret Access Key", runner.createSecretReference("s3-secret-key") + ); + + runner.configure("Kafka Connection", kafkaConnectionConfig, kafkaSecretRefs); + runner.configure("Kafka Topics", kafkaTopicConfig); + runner.configure("S3 Configuration", s3Config, s3SecretRefs); + runner.applyUpdate(); + + final List validationResults = runner.validate(); + assertEquals(Collections.emptyList(), validationResults); + + runner.startConnector(); + try { + runner.waitForDataIngested(Duration.ofSeconds(10)); + runner.waitForIdle(Duration.ofSeconds(30)); + } finally { + runner.stopConnector(); + } + + verifyS3ObjectsCreated(bucketName); + } + + @Test + public void testSwitchFromJsonToAvroS3Format() throws IOException, ExecutionException, InterruptedException, FlowUpdateException { + final String bucketName = "test-switch-formats"; + createS3Bucket(bucketName); + + // Add data to json and avro topics + createKafkaTopics("json", "avro"); + + produceRecordsToTopic("json", + """ + {"id": 1, "type": "json", "data": "First JSON record"}""", + """ + {"id": 2, "type": "json", "data": "Second JSON record"}""", + """ + {"id": 3, "type": "json", "data": "Third JSON record"}""" + ); + + final String schemaString = """ + { + "type": "record", + "name": "AvroRecord", + "namespace": "org.apache.nifi.test", + "fields": [ + {"name": "id", "type": "int"}, + {"name": "type", "type": "string"}, + {"name": "data", "type": "string"} + ] + }"""; + + final Schema schema = new Schema.Parser().parse(schemaString); + + final GenericRecord avroRecord1 = new GenericData.Record(schema); + avroRecord1.put("id", 10); + avroRecord1.put("type", "avro"); + avroRecord1.put("data", "First Avro record"); + + final GenericRecord avroRecord2 = new GenericData.Record(schema); + avroRecord2.put("id", 20); + avroRecord2.put("type", "avro"); + avroRecord2.put("data", "Second Avro record"); + + final GenericRecord avroRecord3 = new GenericData.Record(schema); + avroRecord3.put("id", 30); + avroRecord3.put("type", "avro"); + avroRecord3.put("data", "Third Avro record"); + + produceAvroRecordsToTopic("avro", schema, avroRecord1, avroRecord2, avroRecord3); + + // Configure Connector to consume from JSON Kafka topic and write to S3 in JSON format, but with an invalid S3 endpoint. + // This will cause the data to remain queued, since PutS3Object will fail to write the data. + final Map kafkaServerConfig = Map.of( + "Kafka Brokers", "localhost:9093", + "Security Protocol", "SASL_PLAINTEXT", + "SASL Mechanism", "PLAIN", + "Username", SCRAM_USERNAME + ); + runner.addSecret("kafka-password", SCRAM_PASSWORD); + final Map kafkaSecretRefs = Map.of( + "Password", runner.createSecretReference("kafka-password") + ); + + final Map jsonTopicConfig = Map.of( + "Topic Names", "json", + "Consumer Group ID", "nifi-kafka-to-s3-testReconfiguration", + "Offset Reset", "earliest", + "Kafka Data Format", "JSON" + ); + + final Map s3InvalidConfig = Map.ofEntries( + Map.entry("S3 Region", S3_REGION), + Map.entry("S3 Data Format", "JSON"), + Map.entry("S3 Bucket", bucketName), + Map.entry("S3 Endpoint Override URL", "http://invalid-s3-endpoint:9999"), + Map.entry("S3 Authentication Strategy", "Access Key ID and Secret Key"), + Map.entry("S3 Access Key ID", localStackContainer.getAccessKey()), + Map.entry("Target Object Size", "1 MB"), + Map.entry("Merge Latency", "1 sec") + ); + runner.addSecret("s3-secret-key", localStackContainer.getSecretKey()); + final Map s3SecretRefs = Map.of( + "S3 Secret Access Key", runner.createSecretReference("s3-secret-key") + ); + + runner.configure("Kafka Connection", kafkaServerConfig, kafkaSecretRefs); + runner.configure("Kafka Topics", jsonTopicConfig); + runner.configure("S3 Configuration", s3InvalidConfig, s3SecretRefs); + runner.applyUpdate(); + + // Run the Connector with the invalid S3 endpoint to queue the JSON data. Wait for data to be queued up. + runner.startConnector(); + try { + runner.waitForDataIngested(Duration.ofSeconds(30)); + } finally { + runner.stopConnector(); + } + + // Apply configuration to specify the correct S3 endpoint. Keep S3 Data Format as JSON for now. + // This will allow the Connector to write data to S3 and properly drain the data when we switch the S3 format from JSON to Avro. + final Map s3ValidJsonConfig = Map.ofEntries( + Map.entry("S3 Region", S3_REGION), + Map.entry("S3 Data Format", "JSON"), + Map.entry("S3 Bucket", bucketName), + Map.entry("S3 Endpoint Override URL", localStackContainer.getEndpoint().toString()), + Map.entry("S3 Authentication Strategy", "Access Key ID and Secret Key"), + Map.entry("S3 Access Key ID", localStackContainer.getAccessKey()), + Map.entry("Target Object Size", "1 MB"), + Map.entry("Merge Latency", "1 sec") + ); + + runner.configure("S3 Configuration", s3ValidJsonConfig, s3SecretRefs); + runner.applyUpdate(); + + // Make sure there is no data in S3 yet. + final ListObjectsV2Response initialListingResponse = s3Client.listObjectsV2(ListObjectsV2Request.builder().bucket(bucketName).build()); + final List initialS3Objects = initialListingResponse.contents(); + assertEquals(List.of(), initialS3Objects); + + // Now change the S3 Data Format from JSON to Avro. This should trigger draining of the queued JSON data. + final Map s3ValidAvroConfig = Map.ofEntries( + Map.entry("S3 Region", S3_REGION), + Map.entry("S3 Data Format", "Avro"), + Map.entry("S3 Bucket", bucketName), + Map.entry("S3 Endpoint Override URL", localStackContainer.getEndpoint().toString()), + Map.entry("S3 Authentication Strategy", "Access Key ID and Secret Key"), + Map.entry("S3 Access Key ID", localStackContainer.getAccessKey()), + Map.entry("Target Object Size", "1 MB"), + Map.entry("Merge Latency", "1 sec") + ); + + // Configure to consume from the Avro topic and change S3 format to Avro + final Map avroTopicConfig = Map.of( + "Topic Names", "avro", + "Consumer Group ID", "nifi-kafka-to-s3-testReconfiguration", + "Offset Reset", "earliest", + "Kafka Data Format", "Avro" + ); + + final Map kafkaConnectionWithSchemaRegistry = Map.of( + "Kafka Brokers", "localhost:9093", + "Security Protocol", "SASL_PLAINTEXT", + "SASL Mechanism", "PLAIN", + "Username", SCRAM_USERNAME, + "Schema Registry URL", getSchemaRegistryUrl() + ); + + runner.configure("Kafka Connection", kafkaConnectionWithSchemaRegistry, kafkaSecretRefs); + runner.configure("Kafka Topics", avroTopicConfig); + runner.configure("S3 Configuration", s3ValidAvroConfig, s3SecretRefs); + runner.applyUpdate(); + + // After draining, there should be one JSON file in S3. + final ListObjectsV2Response jsonOnlyListing = s3Client.listObjectsV2(ListObjectsV2Request.builder().bucket(bucketName).build()); + final List jsonObjects = jsonOnlyListing.contents(); + assertEquals(1, jsonObjects.size()); + + // Start connector, wait for more data to be ingested, and then wait for the Connector to be idle. + runner.startConnector(); + try { + runner.waitForDataIngested(Duration.ofSeconds(30)); + runner.waitForIdle(Duration.ofSeconds(30)); + } finally { + runner.stopConnector(); + } + + // Verify that there are two objects in S3: one in JSON format and one in Avro format. + final ListObjectsV2Response listResponse = s3Client.listObjectsV2(ListObjectsV2Request.builder().bucket(bucketName).build()); + final List objects = listResponse.contents(); + + assertEquals(2, objects.size(), "Expected exactly 2 objects in S3 bucket: one for JSON data and one for Avro data"); + + for (final S3Object s3Object : objects) { + final GetObjectRequest getObjectRequest = GetObjectRequest.builder().bucket(bucketName).key(s3Object.key()).build(); + try (final ResponseInputStream objectContent = s3Client.getObject(getObjectRequest)) { + final long objectSize = objectContent.response().contentLength(); + assertTrue(objectSize > 0, "Expected S3 object " + s3Object.key() + " to have content"); + } + } + } + +} diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-nar/pom.xml b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-nar/pom.xml new file mode 100644 index 000000000000..b167c8459532 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/nifi-kafka-to-s3-nar/pom.xml @@ -0,0 +1,36 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi-kafka-to-s3-bundle + 2.9.0-SNAPSHOT + + + nifi-kafka-to-s3-nar + nar + + + + org.apache.nifi + nifi-kafka-to-s3-connector + 2.9.0-SNAPSHOT + + + \ No newline at end of file diff --git a/nifi-connectors/nifi-kafka-to-s3-bundle/pom.xml b/nifi-connectors/nifi-kafka-to-s3-bundle/pom.xml new file mode 100644 index 000000000000..95960d7a0ea4 --- /dev/null +++ b/nifi-connectors/nifi-kafka-to-s3-bundle/pom.xml @@ -0,0 +1,39 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi-connectors + 2.9.0-SNAPSHOT + + + nifi-kafka-to-s3-bundle + pom + + + nifi-kafka-to-s3-connector + nifi-kafka-to-s3-nar + nifi-kafka-to-s3-integration-tests + + + + + + + \ No newline at end of file diff --git a/nifi-connectors/pom.xml b/nifi-connectors/pom.xml new file mode 100644 index 000000000000..8e1b10a447d5 --- /dev/null +++ b/nifi-connectors/pom.xml @@ -0,0 +1,42 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi + 2.9.0-SNAPSHOT + + + nifi-connectors + pom + + + nifi-kafka-to-s3-bundle + + + + + + org.apache.nifi + nifi-connector-utils + 2.9.0-SNAPSHOT + + + + diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index afa152201513..4c4e89b0e3b0 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -1510,6 +1510,11 @@ Global access policies govern the following system level authorizations: |Parameter Contexts |`/parameter-contexts` +|access connectors +|Allows users to view/modify Connectors +|N/A +|`/connectors` + |query provenance |Allows users to submit a Provenance Search and request Event Lineage |Data Provenance diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java index 6ef907fdec22..b4e9beac7001 100644 --- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java +++ b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java @@ -17,7 +17,9 @@ package org.apache.nifi.processors.aws.s3; import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.components.AllowableValue; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.components.ConnectorMethod; import org.apache.nifi.context.PropertyContext; import org.apache.nifi.expression.ExpressionLanguageScope; import org.apache.nifi.flowfile.FlowFile; @@ -26,6 +28,7 @@ import org.apache.nifi.processor.ProcessSession; import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.processors.aws.AbstractAwsSyncProcessor; +import org.apache.nifi.processors.aws.region.RegionUtil; import software.amazon.awssdk.awscore.defaultsmode.DefaultsMode; import software.amazon.awssdk.awscore.exception.AwsErrorDetails; import software.amazon.awssdk.awscore.exception.AwsServiceException; @@ -37,6 +40,7 @@ import software.amazon.awssdk.services.s3.model.ServerSideEncryption; import java.util.Arrays; +import java.util.List; import java.util.Map; import java.util.Optional; import java.util.stream.Collectors; @@ -342,4 +346,11 @@ protected void setEncryptionAttributes(final Map attributes, fin attributes.put(S3_ENCRYPTION_STRATEGY, encryptionService.getStrategyName()); } } + + @ConnectorMethod(name = "getAvailableRegions", description = "Returns the list of available AWS regions") + public List getAvailableRegions() { + return RegionUtil.getAwsRegionAllowableValues().stream() + .map(AllowableValue::getValue) + .toList(); + } } diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-regions/src/main/java/org/apache/nifi/processors/aws/region/RegionUtil.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-regions/src/main/java/org/apache/nifi/processors/aws/region/RegionUtil.java index b5e35d8b3e3d..2626c41c8a10 100644 --- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-regions/src/main/java/org/apache/nifi/processors/aws/region/RegionUtil.java +++ b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-regions/src/main/java/org/apache/nifi/processors/aws/region/RegionUtil.java @@ -100,7 +100,7 @@ private static AllowableValue[] getRegionAllowableValues() { * * @return All available AWS regions as AllowableValues. */ - private static List getAwsRegionAllowableValues() { + public static List getAwsRegionAllowableValues() { final List values = new ArrayList<>(); for (final Region region : Region.regions()) { values.add(createAwsRegionAllowableValue(region)); diff --git a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java index b92bd862a36a..bd79089542c9 100644 --- a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java +++ b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java @@ -26,9 +26,11 @@ import org.apache.nifi.annotation.lifecycle.OnStopped; import org.apache.nifi.components.AllowableValue; import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.PropertyValue; import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.components.ConnectorMethod; import org.apache.nifi.expression.ExpressionLanguageScope; import org.apache.nifi.kafka.processors.common.KafkaUtils; import org.apache.nifi.kafka.processors.consumer.OffsetTracker; @@ -58,11 +60,14 @@ import org.apache.nifi.processor.VerifiableProcessor; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.serialization.RecordReader; import org.apache.nifi.serialization.RecordReaderFactory; import org.apache.nifi.serialization.RecordSetWriterFactory; import org.apache.nifi.util.StringUtils; +import java.io.ByteArrayInputStream; import java.io.IOException; +import java.io.InputStream; import java.nio.charset.Charset; import java.nio.charset.StandardCharsets; import java.time.Duration; @@ -562,26 +567,110 @@ public List verify(final ProcessContext context, final final List verificationResults = new ArrayList<>(); final KafkaConnectionService connectionService = context.getProperty(CONNECTION_SERVICE).asControllerService(KafkaConnectionService.class); - final PollingContext pollingContext = createPollingContext(context); - final KafkaConsumerService consumerService = connectionService.getConsumerService(pollingContext); + final PollingContext pollingContext = createPollingContext(context, null, AutoOffsetReset.EARLIEST); + try (final KafkaConsumerService consumerService = connectionService.getConsumerService(pollingContext)) { + final ConfigVerificationResult partitionVerification = verifyPartitions(consumerService, pollingContext); + verificationResults.add(partitionVerification); + + final ConfigVerificationResult parsingResult = verifyCanParse(context, consumerService, verificationLogger); + verificationResults.add(parsingResult); + } catch (final IOException e) { + verificationResults.add(new ConfigVerificationResult.Builder() + .verificationStepName("Communicate with Kafka Broker") + .outcome(Outcome.FAILED) + .explanation("There was an I/O failure when communicating with Kafka: " + e) + .build()); + } + + return verificationResults; + } - final ConfigVerificationResult.Builder verificationPartitions = new ConfigVerificationResult.Builder() - .verificationStepName("Verify Topic Partitions"); + private ConfigVerificationResult verifyPartitions(final KafkaConsumerService consumerService, final PollingContext pollingContext) { + final ConfigVerificationResult.Builder partitionVerification = new ConfigVerificationResult.Builder() + .verificationStepName("Verify Topic Partitions"); try { final List partitionStates = consumerService.getPartitionStates(); - verificationPartitions - .outcome(ConfigVerificationResult.Outcome.SUCCESSFUL) - .explanation(String.format("Partitions [%d] found for Topics %s", partitionStates.size(), pollingContext.getTopics())); + partitionVerification + .outcome(ConfigVerificationResult.Outcome.SUCCESSFUL) + .explanation(String.format("Found [%d] partitions for Topics %s", partitionStates.size(), pollingContext.getTopics())); } catch (final Exception e) { getLogger().error("Topics {} Partition verification failed", pollingContext.getTopics(), e); - verificationPartitions - .outcome(ConfigVerificationResult.Outcome.FAILED) - .explanation(String.format("Topics %s Partition access failed: %s", pollingContext.getTopics(), e)); + partitionVerification + .outcome(ConfigVerificationResult.Outcome.FAILED) + .explanation(String.format("Topics %s Partition access failed: %s", pollingContext.getTopics(), e)); } - verificationResults.add(verificationPartitions.build()); - return verificationResults; + return partitionVerification.build(); + } + + private ConfigVerificationResult verifyCanParse(final ProcessContext context, final KafkaConsumerService consumerService, final ComponentLog verificationLogger) { + final Iterable records = consumerService.poll(Duration.ofSeconds(60)); + final ProcessingStrategy processingStrategy = context.getProperty(PROCESSING_STRATEGY).asAllowableValue(ProcessingStrategy.class); + if (processingStrategy != ProcessingStrategy.RECORD) { + return new ConfigVerificationResult.Builder() + .verificationStepName("Parse Records") + .outcome(Outcome.SKIPPED) + .explanation("Processing Strategy is set to " + processingStrategy.getValue() + " so skipping record parsing verification") + .build(); + } + + final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class); + int recordIndex = 0; + for (final ByteRecord byteRecord : records) { + recordIndex++; + final Map recordAttributes = KafkaUtils.toAttributes( + byteRecord, keyEncoding, headerNamePattern, headerEncoding, commitOffsets); + + try (final InputStream inputStream = new ByteArrayInputStream(byteRecord.getValue()); + final RecordReader reader = readerFactory.createRecordReader(recordAttributes, inputStream, byteRecord.getValue().length, verificationLogger)) { + + while (reader.nextRecord() != null) { + } + } catch (final Exception e) { + return new ConfigVerificationResult.Builder() + .verificationStepName("Parse Records") + .outcome(Outcome.FAILED) + .explanation("Failed to parse Record number " + recordIndex + ": " + e) + .build(); + } + } + + if (recordIndex == 0) { + return new ConfigVerificationResult.Builder() + .verificationStepName("Parse Records") + .outcome(Outcome.SKIPPED) + .explanation("No records were received to parse") + .build(); + } + + return new ConfigVerificationResult.Builder() + .verificationStepName("Parse Records") + .outcome(Outcome.SUCCESSFUL) + .explanation("Successfully parsed " + recordIndex + " records") + .build(); + } + + + @ConnectorMethod( + name = "sampleTopics", + description = "Returns a list of sample data from the topics that would be consumed by this processor." + ) + public List sampleTopics(final ProcessContext context) throws IOException { + final KafkaConnectionService connectionService = context.getProperty(CONNECTION_SERVICE).asControllerService(KafkaConnectionService.class); + final PollingContext pollingContext = createPollingContext(context, "nifi-validation-" + System.currentTimeMillis(), AutoOffsetReset.EARLIEST); + try (final KafkaConsumerService consumerService = connectionService.getConsumerService(pollingContext)) { + final Iterable records = consumerService.poll(Duration.ofSeconds(60)); + final List samples = new ArrayList<>(); + for (final ByteRecord record : records) { + samples.add(record.getValue()); + if (samples.size() >= 10) { + break; + } + } + + return samples; + } } private KafkaConsumerService getConsumerService(final ProcessContext context) { @@ -670,6 +759,10 @@ private PollingContext createPollingContext(final ProcessContext context) { final String groupId = context.getProperty(GROUP_ID).getValue(); final String offsetReset = context.getProperty(AUTO_OFFSET_RESET).getValue(); final AutoOffsetReset autoOffsetReset = AutoOffsetReset.valueOf(offsetReset.toUpperCase()); + return createPollingContext(context, groupId, autoOffsetReset); + } + + private PollingContext createPollingContext(final ProcessContext context, final String groupId, final AutoOffsetReset autoOffsetReset) { final String topics = context.getProperty(TOPICS).evaluateAttributeExpressions().getValue(); final String topicFormat = context.getProperty(TOPIC_FORMAT).getValue(); @@ -686,4 +779,5 @@ private PollingContext createPollingContext(final ProcessContext context) { return pollingContext; } + } diff --git a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/test/java/org/apache/nifi/kafka/processors/ConsumeKafkaTest.java b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/test/java/org/apache/nifi/kafka/processors/ConsumeKafkaTest.java index 531e5fff0744..14c50d45babe 100644 --- a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/test/java/org/apache/nifi/kafka/processors/ConsumeKafkaTest.java +++ b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/test/java/org/apache/nifi/kafka/processors/ConsumeKafkaTest.java @@ -17,6 +17,7 @@ package org.apache.nifi.kafka.processors; import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; import org.apache.nifi.kafka.service.api.KafkaConnectionService; import org.apache.nifi.kafka.service.api.common.PartitionState; import org.apache.nifi.kafka.service.api.consumer.KafkaConsumerService; @@ -36,6 +37,7 @@ import static org.apache.nifi.kafka.processors.ConsumeKafka.GROUP_ID; import static org.apache.nifi.kafka.processors.ConsumeKafka.TOPICS; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.when; @@ -96,10 +98,15 @@ public void testVerifySuccessful() throws InitializationException { runner.setProperty(GROUP_ID, CONSUMER_GROUP_ID); final List results = processor.verify(runner.getProcessContext(), runner.getLogger(), Collections.emptyMap()); - assertEquals(1, results.size()); + final List successResults = results.stream() + .filter(result -> result.getOutcome() == ConfigVerificationResult.Outcome.SUCCESSFUL) + .toList(); + assertEquals(1, successResults.size()); - final ConfigVerificationResult firstResult = results.iterator().next(); - assertEquals(ConfigVerificationResult.Outcome.SUCCESSFUL, firstResult.getOutcome()); + final boolean anyFailures = results.stream().anyMatch(result -> result.getOutcome() == Outcome.FAILED); + assertFalse(anyFailures, "At least one verification result was a failure: " + results); + + final ConfigVerificationResult firstResult = successResults.getFirst(); assertNotNull(firstResult.getExplanation()); } @@ -113,9 +120,14 @@ public void testVerifyFailed() throws InitializationException { runner.setProperty(GROUP_ID, CONSUMER_GROUP_ID); final List results = processor.verify(runner.getProcessContext(), runner.getLogger(), Collections.emptyMap()); - assertEquals(1, results.size()); + assertEquals(2, results.size()); + + final List failedResults = results.stream() + .filter(result -> result.getOutcome() == ConfigVerificationResult.Outcome.FAILED) + .toList(); + assertEquals(1, failedResults.size()); - final ConfigVerificationResult firstResult = results.iterator().next(); + final ConfigVerificationResult firstResult = failedResults.getFirst(); assertEquals(ConfigVerificationResult.Outcome.FAILED, firstResult.getOutcome()); assertNotNull(firstResult.getExplanation()); } diff --git a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-api/src/main/java/org/apache/nifi/kafka/service/api/consumer/PollingContext.java b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-api/src/main/java/org/apache/nifi/kafka/service/api/consumer/PollingContext.java index 27b07b41e3bc..171ff02fdf9b 100644 --- a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-api/src/main/java/org/apache/nifi/kafka/service/api/consumer/PollingContext.java +++ b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-api/src/main/java/org/apache/nifi/kafka/service/api/consumer/PollingContext.java @@ -34,7 +34,7 @@ public class PollingContext { public PollingContext(final String groupId, final Collection topics, final AutoOffsetReset autoOffsetReset) { - this.groupId = Objects.requireNonNull(groupId, "Group ID required"); + this.groupId = groupId; this.topics = Collections.unmodifiableCollection(Objects.requireNonNull(topics, "Topics required")); this.topicPattern = null; this.autoOffsetReset = Objects.requireNonNull(autoOffsetReset, "Auto Offset Reset required"); @@ -42,7 +42,7 @@ public PollingContext(final String groupId, final Collection topics, public PollingContext(final String groupId, final Pattern topicPattern, final AutoOffsetReset autoOffsetReset) { - this.groupId = Objects.requireNonNull(groupId, "Group ID required"); + this.groupId = groupId; this.topics = Collections.emptyList(); this.topicPattern = Objects.requireNonNull(topicPattern, "Topic Patten required"); this.autoOffsetReset = Objects.requireNonNull(autoOffsetReset, "Auto Offset Reset required"); diff --git a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/Kafka3ConnectionService.java b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/Kafka3ConnectionService.java index a03c0ecac359..c7cc4782ff00 100644 --- a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/Kafka3ConnectionService.java +++ b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/Kafka3ConnectionService.java @@ -16,7 +16,9 @@ */ package org.apache.nifi.kafka.service; +import org.apache.kafka.clients.admin.Admin; import org.apache.kafka.clients.admin.AdminClientConfig; +import org.apache.kafka.clients.admin.ListTopicsResult; import org.apache.kafka.clients.consumer.Consumer; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.KafkaConsumer; @@ -30,6 +32,8 @@ import org.apache.nifi.components.ConfigVerificationResult; import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.connector.components.ConnectorMethod; +import org.apache.nifi.components.connector.components.MethodArgument; import org.apache.nifi.context.PropertyContext; import org.apache.nifi.controller.AbstractControllerService; import org.apache.nifi.controller.ConfigurationContext; @@ -42,6 +46,7 @@ import org.apache.nifi.kafka.service.api.consumer.PollingContext; import org.apache.nifi.kafka.service.api.producer.KafkaProducerService; import org.apache.nifi.kafka.service.api.producer.ProducerConfiguration; +import org.apache.nifi.kafka.service.consumer.Kafka3AssignmentService; import org.apache.nifi.kafka.service.consumer.Kafka3ConsumerService; import org.apache.nifi.kafka.service.consumer.Subscription; import org.apache.nifi.kafka.service.producer.Kafka3ProducerService; @@ -59,11 +64,14 @@ import org.apache.nifi.ssl.SSLContextService; import java.time.Duration; +import java.util.ArrayList; import java.util.List; import java.util.Map; import java.util.Objects; import java.util.Optional; import java.util.Properties; +import java.util.Set; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.regex.Pattern; @@ -235,18 +243,25 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String public KafkaConsumerService getConsumerService(final PollingContext pollingContext) { Objects.requireNonNull(pollingContext, "Polling Context required"); - final Subscription subscription = createSubscription(pollingContext); + final String groupId = pollingContext.getGroupId(); final Properties properties = new Properties(); properties.putAll(consumerProperties); - properties.put(ConsumerConfig.GROUP_ID_CONFIG, subscription.getGroupId()); - properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, subscription.getAutoOffsetReset().getValue()); + if (groupId != null) { + properties.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); + } + properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, pollingContext.getAutoOffsetReset().getValue()); properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); final ByteArrayDeserializer deserializer = new ByteArrayDeserializer(); final Consumer consumer = new KafkaConsumer<>(properties, deserializer, deserializer); - return new Kafka3ConsumerService(getLogger(), consumer, subscription); + if (groupId == null) { + return new Kafka3AssignmentService(consumer, pollingContext.getTopics()); + } else { + final Subscription subscription = createSubscription(pollingContext); + return new Kafka3ConsumerService(getLogger(), consumer, subscription); + } } private Subscription createSubscription(final PollingContext pollingContext) { @@ -291,6 +306,26 @@ public String getBrokerUri() { return uri; } + @ConnectorMethod( + name = "listTopicNames", + description = "Returns a list of topic names available in the Kafka cluster", + arguments = { + @MethodArgument(name = "context", type = ConfigurationContext.class, description = "The configuration context that specifies connectivity details") + }) + public List listTopicNames(final ConfigurationContext context) throws ExecutionException, InterruptedException { + final Properties clientProperties = getClientProperties(context); + final Properties consumerProperties = getConsumerProperties(context, clientProperties); + + try (final Admin admin = Admin.create(consumerProperties)) { + final ListTopicsResult result = admin.listTopics(); + final Set topicNames = result.names().get(); + final List sortedTopicNames = new ArrayList<>(topicNames); + sortedTopicNames.sort(String.CASE_INSENSITIVE_ORDER); + return sortedTopicNames; + } + } + + @Override public List verify(final ConfigurationContext configurationContext, final ComponentLog verificationLogger, final Map variables) { // Build Client Properties based on configured values and defaults from Consumer Properties diff --git a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/Kafka3AssignmentService.java b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/Kafka3AssignmentService.java new file mode 100644 index 000000000000..1fa421d3a2a2 --- /dev/null +++ b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/Kafka3AssignmentService.java @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.kafka.service.consumer; + +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.common.PartitionInfo; +import org.apache.kafka.common.TopicPartition; +import org.apache.nifi.kafka.service.api.common.PartitionState; +import org.apache.nifi.kafka.service.api.common.TopicPartitionSummary; +import org.apache.nifi.kafka.service.api.consumer.KafkaConsumerService; +import org.apache.nifi.kafka.service.api.consumer.PollingSummary; +import org.apache.nifi.kafka.service.api.record.ByteRecord; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.OptionalLong; +import java.util.Set; + +public class Kafka3AssignmentService implements KafkaConsumerService { + private final Consumer consumer; + private volatile boolean closed = false; + + public Kafka3AssignmentService(final Consumer consumer, final Collection topicNames) { + this.consumer = consumer; + + final List topicPartitions = new ArrayList<>(); + for (final String topicName : topicNames) { + final List partitionInfos = consumer.partitionsFor(topicName); + partitionInfos.forEach(info -> topicPartitions.add(new TopicPartition(info.topic(), info.partition()))); + } + + consumer.assign(topicPartitions); + } + + @Override + public void commit(final PollingSummary pollingSummary) { + throw new UnsupportedOperationException("Commit not supported for assignment-based consumer"); + } + + @Override + public void rollback() { + // no-op + } + + @Override + public boolean isClosed() { + return false; + } + + @Override + public Iterable poll(final Duration maxWaitDuration) { + final ConsumerRecords consumerRecords = consumer.poll(maxWaitDuration); + if (consumerRecords.isEmpty()) { + return List.of(); + } + + return new RecordIterable(consumerRecords); + } + + @Override + public List getPartitionStates() { + final Set topicPartitions = consumer.assignment(); + return topicPartitions.stream() + .map(tp -> new PartitionState(tp.topic(), tp.partition())) + .toList(); + } + + @Override + public OptionalLong currentLag(final TopicPartitionSummary topicPartitionSummary) { + return OptionalLong.empty(); + } + + @Override + public void close() { + if (closed) { + return; + } + + closed = true; + consumer.close(); + } +} diff --git a/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/RecordIterable.java b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/RecordIterable.java new file mode 100644 index 000000000000..6f8229eb2f81 --- /dev/null +++ b/nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/RecordIterable.java @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.kafka.service.consumer; + +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.nifi.kafka.service.api.header.RecordHeader; +import org.apache.nifi.kafka.service.api.record.ByteRecord; + +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +public class RecordIterable implements Iterable { + private final Iterator records; + + public RecordIterable(final Iterable> consumerRecords) { + this.records = new RecordIterator(consumerRecords); + } + + @Override + public Iterator iterator() { + return records; + } + + private static class RecordIterator implements Iterator { + private final Iterator> consumerRecords; + + private RecordIterator(final Iterable> records) { + this.consumerRecords = records.iterator(); + } + + @Override + public boolean hasNext() { + return consumerRecords.hasNext(); + } + + @Override + public ByteRecord next() { + final ConsumerRecord consumerRecord = consumerRecords.next(); + final List recordHeaders = new ArrayList<>(); + consumerRecord.headers().forEach(header -> { + final RecordHeader recordHeader = new RecordHeader(header.key(), header.value()); + recordHeaders.add(recordHeader); + }); + + // Support Kafka tombstones + byte[] value = consumerRecord.value(); + if (value == null) { + value = new byte[0]; + } + + return new ByteRecord( + consumerRecord.topic(), + consumerRecord.partition(), + consumerRecord.offset(), + consumerRecord.timestamp(), + recordHeaders, + consumerRecord.key(), + value, + 1 + ); + } + } +} diff --git a/nifi-extension-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/script/impl/ValidationContextAdapter.java b/nifi-extension-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/script/impl/ValidationContextAdapter.java index d78d01347820..8951412fc005 100644 --- a/nifi-extension-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/script/impl/ValidationContextAdapter.java +++ b/nifi-extension-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/script/impl/ValidationContextAdapter.java @@ -115,4 +115,9 @@ public Collection getReferencedParameters(final String propertyName) { public boolean isDependencySatisfied(final PropertyDescriptor propertyDescriptor, final Function propertyDescriptorLookup) { return innerValidationContext.isDependencySatisfied(propertyDescriptor, propertyDescriptorLookup); } + + @Override + public String evaluateParameters(final String value) { + return innerValidationContext.evaluateParameters(value); + } } diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java index 3297dc621744..94f8b0ac1d46 100644 --- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java +++ b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DuplicateFlowFile.java @@ -43,7 +43,7 @@ + "generated copies are sent to the 'success' relationship. In addition, each FlowFile gets an attribute 'copy.index' set to the copy number, where the original FlowFile gets " + "a value of zero, and all copies receive incremented integer values.") @WritesAttributes({ - @WritesAttribute(attribute = "copy.index", description = "A zero-based incrementing integer value based on which copy the FlowFile is.") + @WritesAttribute(attribute = "copy.index", description = "A zero-based incrementing integer value based on which copy the FlowFile is.") }) public class DuplicateFlowFile extends AbstractProcessor { diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/asset/AssetManager.java b/nifi-framework-api/src/main/java/org/apache/nifi/asset/AssetManager.java index d1501c2f9335..321d0afe7b9f 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/asset/AssetManager.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/asset/AssetManager.java @@ -31,13 +31,25 @@ public interface AssetManager { /** * Creates a new Asset with the given name and contents. - * @param parameterContextId the id of the parameter context + * @param ownerId the id of the resource that this asset belongs to * @param assetName the name of the asset * @param contents the contents of the asset * @return the created asset * @throws IOException if there is an error creating the asset */ - Asset createAsset(String parameterContextId, String assetName, InputStream contents) throws IOException; + Asset createAsset(String ownerId, String assetName, InputStream contents) throws IOException; + + /** + * Saves the given asset. If an asset already exists with the given identifier, the content will be replaced. + * + * @param ownerId the id of the resource that this asset belongs to + * @param assetId the identifier of the asset + * @param assetName the name of the asset + * @param contents the new contents of the asset + * @return the created or updated Asset + * @throws IOException if there is an error writing the asset + */ + Asset saveAsset(String ownerId, String assetId, String assetName, InputStream contents) throws IOException; /** * Retrieves the Asset with the given id, if it exists. @@ -47,22 +59,22 @@ public interface AssetManager { Optional getAsset(String id); /** - * Retrieves the Assets that belong to the given parameter context. - * @param parameterContextId the id of the parameter context - * @return the list of assets for the given context + * Retrieves the Assets that belong to the given owner. + * @param ownerId the id of the owner resource + * @return the list of assets for the given owner */ - List getAssets(String parameterContextId); + List getAssets(String ownerId); /** - * Creates an Asset with the given name and associates it with the given parameter context. If the asset already exists, it is returned. Otherwise, an asset is created + * Creates an Asset with the given name and associates it with the given owner. If the asset already exists, it is returned. Otherwise, an asset is created * but the underlying file is not created. This allows the asset to be referenced but any component that attempts to use the asset will still see a File that does not exist, which * will typically lead to an invalid component. * - * @param parameterContextId the id of the parameter context + * @param ownerId the id of the owner * @param assetName the name of the asset * @return the created asset */ - Asset createMissingAsset(String parameterContextId, String assetName); + Asset createMissingAsset(String ownerId, String assetName); /** * Deletes the Asset with the given id, if it exists. diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProvider.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProvider.java new file mode 100644 index 000000000000..4d2ff3ecc88c --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProvider.java @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.io.InputStream; +import java.util.Optional; + +/** + * Extension point interface for external management of Connector working configuration and assets. + * Implementations allow a Connector's name, working flow configuration, and binary assets to be + * persisted in an external store and to be externally modified. + * + *

When a ConnectorConfigurationProvider is configured, the framework will: + *

    + *
  • On reads (e.g., getConnector): load configuration from the provider and override the in-memory working configuration
    • + *
  • On writes (e.g., configureConnector): save configuration to the provider before modifying the in-memory state
    • + *
  • On discard: notify the provider that the working configuration has been discarded
    • + *
  • On delete: notify the provider that the connector has been removed
    • + *
+ * + *

Asset management: the provider owns the mapping between NiFi asset UUIDs and external asset + * identifiers. The {@link #load} method must return {@link ConnectorWorkingConfiguration} with + * NiFi UUIDs in {@code assetIds} fields. The {@link #save} method receives NiFi UUIDs and the + * provider translates to external identifiers when persisting. The local state of assets + * (NiFi UUID mapping and digest tracking) is managed by the provider using the + * {@link org.apache.nifi.asset.AssetManager} provided via the initialization context.

+ */ +public interface ConnectorConfigurationProvider { + + /** + * Initializes the ConnectorConfigurationProvider with the given context. + * + * @param context the initialization context providing configuration properties + */ + void initialize(ConnectorConfigurationProviderInitializationContext context); + + /** + * Loads the externally managed working configuration for the connector with the given identifier. + * + * @param connectorId the identifier of the connector + * @return an Optional containing the working configuration if one exists in the external store, + * or an empty Optional if no external configuration exists (in which case the in-memory configuration is used) + */ + Optional load(String connectorId); + + /** + * Saves the working configuration for the connector with the given identifier to the external store. + * This is called when working configuration properties are modified, such as during configureConnector + * or when connector metadata (e.g., name) is updated. + * + * @param connectorId the identifier of the connector + * @param configuration the working configuration to save + */ + void save(String connectorId, ConnectorWorkingConfiguration configuration); + + /** + * Notifies the provider that the working configuration for the given connector has been discarded + * (i.e., reset to match the active configuration). This is semantically distinct from {@link #save} + * because the external store may need to handle this differently, such as deleting the working copy + * rather than overwriting it. + * + * @param connectorId the identifier of the connector whose working configuration was discarded + */ + void discard(String connectorId); + + /** + * Notifies the provider that the connector with the given identifier has been removed entirely. + * The provider should clean up any stored configuration for this connector. + * + * @param connectorId the identifier of the connector that was removed + */ + void delete(String connectorId); + + /** + * Verifies that the provider can support creating a connector with the given identifier. + * This is called before the connector is actually created, giving the provider an opportunity + * to reject the operation (for example, if it has reached a capacity limit or the connector + * already exists in the external store in an incompatible state). + * + *

If the provider cannot support the create operation, it should throw a + * {@link ConnectorConfigurationProviderException}.

+ * + * @param connectorId the identifier of the connector to be created + */ + void verifyCreate(String connectorId); + + /** + * Stores an asset to the local {@link org.apache.nifi.asset.AssetManager} and to the external + * store. The provider records the NiFi UUID to external identifier mapping in its local state. + * If the external store upload fails, the provider must roll back the local asset and throw. + * + * @param connectorId the identifier of the connector that owns the asset + * @param nifiUuid the NiFi-assigned UUID for this asset + * @param assetName the filename of the asset (e.g., "postgresql-42.6.0.jar") + * @param content the binary content of the asset + * @throws java.io.IOException if the asset cannot be stored + */ + void storeAsset(String connectorId, String nifiUuid, String assetName, InputStream content) throws java.io.IOException; + + /** + * Deletes an asset from the local {@link org.apache.nifi.asset.AssetManager} and from the + * external store. The provider uses the NiFi UUID to look up the external identifier from its + * local state, then cleans up both stores and removes the mapping entry. + * + * @param connectorId the identifier of the connector that owns the asset + * @param nifiUuid the NiFi-assigned UUID of the asset to delete + */ + void deleteAsset(String connectorId, String nifiUuid); + + /** + * Called when a connector update is requested (e.g., applying a committed configuration change) + * to determine whether the framework should proceed with the standard internal update process + * (stopping, re-configuring, and restarting the connector). + * + *

Returning {@code true} (the default) indicates the framework should proceed normally.

+ * + *

Returning {@code false} indicates the framework should skip the update and return + * immediately -- this is not a failure; the provider may have handled the update externally + * by doing some bookkeeping logic and the provider may re-trigger the update process by starting + * a new request to the nifi framework once it is ready to proceed. If the provider wants to fail + * the request, it should throw a runtime exception instead.

+ * + * @param connectorId the identifier of the connector to update + * @return {@code true} if the framework should proceed with the standard update process, + * {@code false} if the framework should skip the update (no-op) + */ + default boolean shouldApplyUpdate(final String connectorId) { + return true; + } + + /** + * Ensures that local asset binaries are up to date with the external store. For each asset + * tracked in the provider's local state, this method compares the external store's current + * content digest to the last-known digest. If changed or missing locally, the binary is + * downloaded via the {@link org.apache.nifi.asset.AssetManager} using a new UUID so that + * any existing local file for a running connector is not overwritten. The local state file + * is updated with the new UUID and digest. This method does not modify the external store's + * configuration. + * + *

After this method returns, callers should invoke {@link #load} to obtain the updated + * working configuration reflecting any new NiFi UUIDs assigned during sync.

+ * + * @param connectorId the identifier of the connector whose assets should be synced + */ + void syncAssets(String connectorId); +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProviderException.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProviderException.java new file mode 100644 index 000000000000..de772d0a17e7 --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProviderException.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +/** + * Runtime exception thrown by {@link ConnectorConfigurationProvider} implementations + * to indicate a failure in an external configuration operation such as load, save, discard, or delete. + * + *

This exception type allows provider implementations to signal failures using a well-defined + * exception rather than generic exceptions like IOException or database-specific exceptions. + * The framework will propagate these exceptions to callers rather than handling them silently.

+ */ +public class ConnectorConfigurationProviderException extends RuntimeException { + + public ConnectorConfigurationProviderException(final String message) { + super(message); + } + + public ConnectorConfigurationProviderException(final String message, final Throwable cause) { + super(message, cause); + } +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProviderInitializationContext.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProviderInitializationContext.java new file mode 100644 index 000000000000..bcf7dce26c4c --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfigurationProviderInitializationContext.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; + +import java.util.Map; + +/** + * Initialization context for a {@link ConnectorConfigurationProvider}. + * Provides the configuration properties necessary for the provider to initialize itself, + * such as database connection strings or other external store configuration. + * + *

Properties are extracted from NiFi properties with the prefix + * {@code nifi.components.connectors.configuration.provider.} stripped. For example, + * a NiFi property {@code nifi.components.connectors.configuration.provider.db.url=jdbc:...} + * becomes {@code db.url=jdbc:...} in the returned map.

+ */ +public interface ConnectorConfigurationProviderInitializationContext { + + /** + * Returns the configuration properties for this provider. + * + * @return a map of property names to values + */ + Map getProperties(); + + /** + * Returns the {@link AssetManager} that the provider should use for local asset storage. + * The provider uses this to create, save, and delete asset files on the local NiFi filesystem. + * + * @return the AssetManager for connector assets + */ + AssetManager getAssetManager(); +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestContext.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestContext.java new file mode 100644 index 000000000000..a722abac257a --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestContext.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.authorization.user.NiFiUser; + +import java.util.List; +import java.util.Map; + +/** + * Provides request-scoped context from an HTTP request to a {@link ConnectorConfigurationProvider}. + * + *

When a connector operation is triggered by an HTTP request (e.g., from the Runtime UI), + * the framework populates this context with the authenticated NiFi user and all HTTP headers + * from the original request. Provider implementations can use this to make decisions based on + * who is making the request and to extract forwarded credentials (e.g., OAuth tokens relayed + * by a gateway).

+ * + *

This context is available via {@link ConnectorRequestContextHolder#getContext()} on the + * request thread. When no HTTP request is in scope (e.g., background operations), the holder + * returns {@code null}.

+ */ +public interface ConnectorRequestContext { + + /** + * Returns the authenticated NiFi user who initiated the request. For proxied requests + * (e.g., through a gateway), this is the end-user with the proxy chain accessible + * via {@link NiFiUser#getChain()}. + * + * @return the authenticated NiFi user, or {@code null} if not available + */ + NiFiUser getAuthenticatedUser(); + + /** + * Returns all HTTP headers from the original request as an immutable, case-insensitive + * multi-valued map. Each header name maps to an unmodifiable list of its values. + * + *

Provider implementations should read the specific headers they need and ignore + * the rest. This allows new headers to be forwarded by a gateway without requiring + * changes to the NiFi framework.

+ * + * @return an immutable map of header names to their values + */ + Map> getRequestHeaders(); + + /** + * Returns whether the request contains a header with the given name. + * Header name matching is case-insensitive per the HTTP specification. + * + * @param headerName the header name to check + * @return {@code true} if the header is present, {@code false} otherwise + */ + boolean hasRequestHeader(String headerName); + + /** + * Returns all values for the given header name, or an empty list if the header is not present. + * Header name matching is case-insensitive per the HTTP specification. + * + * @param headerName the header name to look up + * @return an unmodifiable list of header values, or an empty list if not present + */ + List getRequestHeaderValues(String headerName); + + /** + * Returns the first value for the given header name, or {@code null} if the header is + * not present or has no values. Header name matching is case-insensitive per the HTTP + * specification. + * + * @param headerName the header name to look up + * @return the first header value, or {@code null} if not present + */ + String getFirstRequestHeaderValue(String headerName); +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestContextHolder.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestContextHolder.java new file mode 100644 index 000000000000..db46ec53a303 --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestContextHolder.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +/** + * Thread-local holder for {@link ConnectorRequestContext}. This class resides in + * {@code nifi-framework-api} so that it is on the shared classloader, accessible + * from both the NiFi web layer (which sets the context) and NAR-loaded + * {@link ConnectorConfigurationProvider} implementations (which read it). + * + *

The context is set by the framework's request filter before connector operations + * and cleared after the request completes. Provider implementations should access the + * context via {@link #getContext()} and must not assume it is always present -- it will + * be {@code null} for operations not triggered by an HTTP request (e.g., background tasks).

+ */ +public final class ConnectorRequestContextHolder { + + private static final ThreadLocal CONTEXT = new ThreadLocal<>(); + + private ConnectorRequestContextHolder() { + } + + /** + * Returns the {@link ConnectorRequestContext} for the current thread, or {@code null} + * if no context has been set (e.g., for background or non-HTTP-request operations). + * + * @return the current request context, or {@code null} + */ + public static ConnectorRequestContext getContext() { + return CONTEXT.get(); + } + + /** + * Sets the {@link ConnectorRequestContext} for the current thread. + * + * @param context the request context to set + */ + public static void setContext(final ConnectorRequestContext context) { + CONTEXT.set(context); + } + + /** + * Clears the {@link ConnectorRequestContext} from the current thread. + * This must be called after request processing completes to prevent memory leaks. + */ + public static void clearContext() { + CONTEXT.remove(); + } +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorWorkingConfiguration.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorWorkingConfiguration.java new file mode 100644 index 000000000000..3d38d74c90ec --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/ConnectorWorkingConfiguration.java @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.flow.VersionedConfigurationStep; + +import java.util.List; + +/** + * Represents the externally managed working configuration of a Connector, + * including its name and working flow configuration steps. + * + *

This is a mutable POJO used as the input/output for {@link ConnectorConfigurationProvider} + * operations. The style follows the same pattern as {@link VersionedConfigurationStep} and other + * versioned types in the NiFi API.

+ * + *

Asset references in {@code workingFlowConfiguration} use NiFi UUIDs in the {@code assetIds} + * fields. The {@link ConnectorConfigurationProvider} is responsible for translating between NiFi + * UUIDs and any external asset identifiers used by the external store.

+ */ +public class ConnectorWorkingConfiguration { + private String name; + private List workingFlowConfiguration; + + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + public List getWorkingFlowConfiguration() { + return workingFlowConfiguration; + } + + public void setWorkingFlowConfiguration(final List workingFlowConfiguration) { + this.workingFlowConfiguration = workingFlowConfiguration; + } +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/StandardConnectorRequestContext.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/StandardConnectorRequestContext.java new file mode 100644 index 000000000000..97a7e006019f --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/StandardConnectorRequestContext.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.authorization.user.NiFiUser; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.TreeMap; + +/** + * Standard implementation of {@link ConnectorRequestContext} that stores an authenticated + * {@link NiFiUser} and HTTP headers in a case-insensitive map. + * + *

Header name lookups via {@link #hasRequestHeader}, {@link #getRequestHeaderValues}, and + * {@link #getFirstRequestHeaderValue} are case-insensitive per the HTTP specification. + * The backing map uses {@link String#CASE_INSENSITIVE_ORDER} to guarantee this.

+ */ +public class StandardConnectorRequestContext implements ConnectorRequestContext { + + private final NiFiUser niFiUser; + private final Map> requestHeaders; + + /** + * Creates a new context with the given user and headers. The provided headers map is + * copied into a case-insensitive map; the original map is not retained. + * + * @param niFiUser the authenticated NiFi user, or {@code null} if not available + * @param requestHeaders the HTTP headers from the request; may be {@code null} or empty + */ + public StandardConnectorRequestContext(final NiFiUser niFiUser, final Map> requestHeaders) { + this.niFiUser = niFiUser; + if (requestHeaders == null || requestHeaders.isEmpty()) { + this.requestHeaders = Map.of(); + } else { + final Map> caseInsensitive = new TreeMap<>(String.CASE_INSENSITIVE_ORDER); + caseInsensitive.putAll(requestHeaders); + this.requestHeaders = Collections.unmodifiableMap(caseInsensitive); + } + } + + @Override + public NiFiUser getAuthenticatedUser() { + return niFiUser; + } + + @Override + public Map> getRequestHeaders() { + return requestHeaders; + } + + @Override + public boolean hasRequestHeader(final String headerName) { + return requestHeaders.containsKey(headerName); + } + + @Override + public List getRequestHeaderValues(final String headerName) { + final List values = requestHeaders.get(headerName); + return values != null ? values : List.of(); + } + + @Override + public String getFirstRequestHeaderValue(final String headerName) { + final List values = getRequestHeaderValues(headerName); + return values.isEmpty() ? null : values.getFirst(); + } +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/secrets/AuthorizableSecret.java b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/secrets/AuthorizableSecret.java new file mode 100644 index 000000000000..3bcf6b099cd1 --- /dev/null +++ b/nifi-framework-api/src/main/java/org/apache/nifi/components/connector/secrets/AuthorizableSecret.java @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.components.connector.Secret; + +public interface AuthorizableSecret extends Secret, Authorizable { + +} diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileStatus.java b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileStatus.java index 737fbe3884f5..13d25bb2ed1a 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileStatus.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileStatus.java @@ -17,6 +17,8 @@ package org.apache.nifi.controller.queue; +import java.util.concurrent.CompletableFuture; + /** * Represents the status of a Drop FlowFile Request that has been issued to * a {@link FlowFileQueue}. When a queue is requested to drop its FlowFiles, @@ -75,4 +77,9 @@ public interface DropFlowFileStatus { * @return the reason that the state is set to a Failure state, or null if the state is not {@link DropFlowFileState#FAILURE}. */ String getFailureReason(); + + /** + * @return a Future that can be used to determine when the drop operation has completed + */ + CompletableFuture getCompletionFuture(); } diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileQueue.java b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileQueue.java index 6911fd7853d8..2a0f4d6895cb 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileQueue.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileQueue.java @@ -16,9 +16,11 @@ */ package org.apache.nifi.controller.queue; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.controller.status.FlowFileAvailability; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.processor.FlowFileFilter; @@ -27,6 +29,7 @@ import java.util.List; import java.util.Set; import java.util.concurrent.TimeUnit; +import java.util.function.Predicate; public interface FlowFileQueue { @@ -217,6 +220,18 @@ public interface FlowFileQueue { */ DropFlowFileStatus cancelDropFlowFileRequest(String requestIdentifier); + /** + * Synchronously drops all FlowFiles in this queue that match the given predicate. This method filters + * FlowFiles in the active queue, swap queue, and any swapped-out swap files. The FlowFile Repository + * and Provenance Repository are updated atomically after all matching FlowFiles have been identified. + * + * @param predicate the predicate used to determine which FlowFiles should be dropped; FlowFiles for which + * the predicate returns true will be dropped + * @return a summary of the FlowFiles that were dropped, including the count and total size in bytes + * @throws IOException if an error occurs while reading or writing swap files + */ + DropFlowFileSummary dropFlowFiles(Predicate predicate) throws IOException; + /** *

* Initiates a request to obtain a listing of FlowFiles in this queue. This method returns a diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/LocalQueuePartitionDiagnostics.java b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/LocalQueuePartitionDiagnostics.java index cc097b1a52e9..87ae47a4ee6e 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/LocalQueuePartitionDiagnostics.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/LocalQueuePartitionDiagnostics.java @@ -29,4 +29,14 @@ public interface LocalQueuePartitionDiagnostics { boolean isAnyActiveFlowFilePenalized(); boolean isAllActiveFlowFilesPenalized(); + + /** + * @return the QueueSize representing the penalized FlowFiles (count and total content size) + */ + QueueSize getPenalizedQueueSize(); + + /** + * @return the total QueueSize across all swap files (count and total content size) + */ + QueueSize getTotalSwapFileQueueSize(); } diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/FlowFileSwapManager.java b/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/FlowFileSwapManager.java index 018f27425a58..b29bb8b38024 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/FlowFileSwapManager.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/FlowFileSwapManager.java @@ -126,6 +126,16 @@ public interface FlowFileSwapManager { */ void purge(); + /** + * Deletes the swap file at the given location without updating the FlowFile Repository. + * This method is intended for use after the FlowFile Repository has already been updated + * to reflect that the FlowFiles are no longer in this swap file. + * + * @param swapLocation the location of the swap file to delete + * @throws IOException if unable to delete the swap file + */ + void deleteSwapFile(String swapLocation) throws IOException; + /** * Returns the ID of the queue that the given swap file belongs to * @param swapLocation the swap location diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecord.java b/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecord.java index 7a56f2bfce3b..62b5495e3c0a 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecord.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecord.java @@ -82,6 +82,14 @@ public interface RepositoryRecord { */ String getSwapLocation(); + /** + * @return For SWAP_FILE_RENAMED records, provides the original swap location before the rename. + * For other record types, returns null. + */ + default String getOriginalSwapLocation() { + return null; + } + /** * @return a List of Content Claims that are "transient," meaning that they existed only for the * life of the Process Session in which they were created and should not be persisted. diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordType.java b/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordType.java index 50221bba6a8a..af419fa40772 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordType.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordType.java @@ -21,5 +21,5 @@ */ public enum RepositoryRecordType { - UPDATE, CREATE, DELETE, CONTENTMISSING, SWAP_IN, SWAP_OUT, CLEANUP_TRANSIENT_CLAIMS; + UPDATE, CREATE, DELETE, CONTENTMISSING, SWAP_IN, SWAP_OUT, CLEANUP_TRANSIENT_CLAIMS, SWAP_FILE_DELETED, SWAP_FILE_RENAMED; } diff --git a/nifi-framework-api/src/main/java/org/apache/nifi/web/UiExtensionType.java b/nifi-framework-api/src/main/java/org/apache/nifi/web/UiExtensionType.java index 17c2d91e2df5..3cde479f0fc7 100644 --- a/nifi-framework-api/src/main/java/org/apache/nifi/web/UiExtensionType.java +++ b/nifi-framework-api/src/main/java/org/apache/nifi/web/UiExtensionType.java @@ -31,4 +31,5 @@ public enum UiExtensionType { ReportingTaskConfiguration, ParameterProviderConfiguration, FlowRegistryClientConfiguration, + Connector, } diff --git a/nifi-framework-api/src/test/java/org/apache/nifi/components/connector/ConnectorRequestContextHolderTest.java b/nifi-framework-api/src/test/java/org/apache/nifi/components/connector/ConnectorRequestContextHolderTest.java new file mode 100644 index 000000000000..ce9cba3eb672 --- /dev/null +++ b/nifi-framework-api/src/test/java/org/apache/nifi/components/connector/ConnectorRequestContextHolderTest.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.components.connector; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; + +class ConnectorRequestContextHolderTest { + + @AfterEach + void clearContext() { + ConnectorRequestContextHolder.clearContext(); + } + + @Test + void testGetContextReturnsNullByDefault() { + assertNull(ConnectorRequestContextHolder.getContext()); + } + + @Test + void testSetAndGetContext() { + final ConnectorRequestContext context = new StandardConnectorRequestContext(null, Map.of()); + ConnectorRequestContextHolder.setContext(context); + + assertEquals(context, ConnectorRequestContextHolder.getContext()); + } + + @Test + void testClearContextRemovesContext() { + final ConnectorRequestContext context = new StandardConnectorRequestContext(null, Map.of()); + ConnectorRequestContextHolder.setContext(context); + ConnectorRequestContextHolder.clearContext(); + + assertNull(ConnectorRequestContextHolder.getContext()); + } + + @Test + void testContextIsThreadLocal() throws Exception { + final ConnectorRequestContext context = new StandardConnectorRequestContext(null, Map.of()); + ConnectorRequestContextHolder.setContext(context); + + final ConnectorRequestContext[] otherThreadContext = new ConnectorRequestContext[1]; + final Thread thread = new Thread(() -> otherThreadContext[0] = ConnectorRequestContextHolder.getContext()); + thread.start(); + thread.join(); + + assertNull(otherThreadContext[0]); + assertEquals(context, ConnectorRequestContextHolder.getContext()); + } +} diff --git a/nifi-framework-bundle/nifi-framework-extensions/nifi-provenance-repository-bundle/nifi-volatile-provenance-repository/src/test/java/org/apache/nifi/provenance/TestVolatileProvenanceRepository.java b/nifi-framework-bundle/nifi-framework-extensions/nifi-provenance-repository-bundle/nifi-volatile-provenance-repository/src/test/java/org/apache/nifi/provenance/TestVolatileProvenanceRepository.java index 5680d3c3306e..e02b8fba8553 100644 --- a/nifi-framework-bundle/nifi-framework-extensions/nifi-provenance-repository-bundle/nifi-volatile-provenance-repository/src/test/java/org/apache/nifi/provenance/TestVolatileProvenanceRepository.java +++ b/nifi-framework-bundle/nifi-framework-extensions/nifi-provenance-repository-bundle/nifi-volatile-provenance-repository/src/test/java/org/apache/nifi/provenance/TestVolatileProvenanceRepository.java @@ -48,7 +48,7 @@ public static void setup() { @Test public void testAddAndGet() throws IOException { - repo = new VolatileProvenanceRepository(NiFiProperties.createBasicNiFiProperties(null)); + repo = new VolatileProvenanceRepository(NiFiProperties.createBasicNiFiProperties((String) null)); final Map attributes = new HashMap<>(); attributes.put("abc", "xyz"); @@ -81,7 +81,7 @@ public void testAddAndGet() throws IOException { @Test public void testIndexAndCompressOnRolloverAndSubsequentSearchAsync() throws InterruptedException { - repo = new VolatileProvenanceRepository(NiFiProperties.createBasicNiFiProperties(null)); + repo = new VolatileProvenanceRepository(NiFiProperties.createBasicNiFiProperties((String) null)); final String uuid = "00000000-0000-0000-0000-000000000000"; final Map attributes = new HashMap<>(); @@ -120,7 +120,7 @@ public void testIndexAndCompressOnRolloverAndSubsequentSearchAsync() throws Inte @Test public void testSearchForInverseValue() throws InterruptedException { - repo = new VolatileProvenanceRepository(NiFiProperties.createBasicNiFiProperties(null)); + repo = new VolatileProvenanceRepository(NiFiProperties.createBasicNiFiProperties((String) null)); final Map attributes = new HashMap<>(); attributes.put("abc", "xyz"); diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigVerificationResultDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigVerificationResultDTO.java index 13841ca76f61..8134edc5036f 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigVerificationResultDTO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigVerificationResultDTO.java @@ -22,6 +22,7 @@ public class ConfigVerificationResultDTO { private String outcome; private String verificationStepName; + private String subject; private String explanation; @Schema(description = "The outcome of the verification", allowableValues = {"SUCCESSFUL", "FAILED", "SKIPPED"}) @@ -42,6 +43,15 @@ public void setVerificationStepName(final String verificationStepName) { this.verificationStepName = verificationStepName; } + @Schema(description = "The subject of the verification step") + public String getSubject() { + return subject; + } + + public void setSubject(final String subject) { + this.subject = subject; + } + @Schema(description = "An explanation of why the step was or was not successful") public String getExplanation() { return explanation; diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigurationStepConfigurationDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigurationStepConfigurationDTO.java new file mode 100644 index 000000000000..402ecb60a00c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigurationStepConfigurationDTO.java @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.List; +import java.util.Set; + +/** + * The configuration for a configuration step. + */ +@XmlType(name = "configurationStepConfiguration") +public class ConfigurationStepConfigurationDTO { + + private String configurationStepName; + private String configurationStepDescription; + private boolean documented; + private List propertyGroupConfigurations; + private Set dependencies; + + /** + * @return the configuration step name + */ + @Schema(description = "The name of the configuration step.") + public String getConfigurationStepName() { + return configurationStepName; + } + + public void setConfigurationStepName(final String configurationStepName) { + this.configurationStepName = configurationStepName; + } + + /** + * @return the configuration step description + */ + @Schema(description = "The description of the configuration step.") + public String getConfigurationStepDescription() { + return configurationStepDescription; + } + + public void setConfigurationStepDescription(final String configurationStepDescription) { + this.configurationStepDescription = configurationStepDescription; + } + + /** + * @return whether this step has extended documentation available + */ + @Schema(description = "Whether extended documentation is available for this configuration step.") + public boolean isDocumented() { + return documented; + } + + public void setDocumented(final boolean documented) { + this.documented = documented; + } + + /** + * @return the property group configurations + */ + @Schema(description = "The list of property group configurations for this configuration step.") + public List getPropertyGroupConfigurations() { + return propertyGroupConfigurations; + } + + public void setPropertyGroupConfigurations(final List propertyGroupConfigurations) { + this.propertyGroupConfigurations = propertyGroupConfigurations; + } + + /** + * @return the dependencies that this configuration step has on other configuration steps' properties + */ + @Schema(description = "The dependencies that this configuration step has on other configuration steps' properties.") + public Set getDependencies() { + return dependencies; + } + + public void setDependencies(final Set dependencies) { + this.dependencies = dependencies; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigurationStepDependencyDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigurationStepDependencyDTO.java new file mode 100644 index 000000000000..ee59271cfe95 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConfigurationStepDependencyDTO.java @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.Set; + +/** + * A dependency that a configuration step has on another configuration step's property. + */ +@XmlType(name = "configurationStepDependency") +public class ConfigurationStepDependencyDTO { + + private String stepName; + private String propertyName; + private Set dependentValues; + + /** + * @return the name of the configuration step that this step depends on + */ + @Schema(description = "The name of the configuration step that this step depends on.") + public String getStepName() { + return stepName; + } + + public void setStepName(final String stepName) { + this.stepName = stepName; + } + + /** + * @return the name of the property within the dependent step that must have a value + */ + @Schema(description = "The name of the property within the dependent step that must have a value.") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(final String propertyName) { + this.propertyName = propertyName; + } + + /** + * @return the values of the dependent property that satisfy this dependency, or null if any non-null value satisfies the dependency + */ + @Schema(description = "The values of the dependent property that satisfy this dependency. If null, any non-null value satisfies the dependency.") + public Set getDependentValues() { + return dependentValues; + } + + public void setDependentValues(final Set dependentValues) { + this.dependentValues = dependentValues; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorActionDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorActionDTO.java new file mode 100644 index 000000000000..97e27fbfcb7e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorActionDTO.java @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +/** + * Represents an action that can be performed on a Connector. + */ +@XmlType(name = "connectorAction") +public class ConnectorActionDTO { + + private String name; + private String description; + private Boolean allowed; + private String reasonNotAllowed; + + @Schema(description = "The name of the action.") + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + @Schema(description = "A description of what this action does.") + public String getDescription() { + return description; + } + + public void setDescription(final String description) { + this.description = description; + } + + @Schema(description = "Whether this action is currently allowed to be performed, based on the state of the Connector. " + + "Note that a value of 'true' does not imply that the user has permission to perform the action.") + public Boolean getAllowed() { + return allowed; + } + + public void setAllowed(final Boolean allowed) { + this.allowed = allowed; + } + + @Schema(description = "The reason why this action is not allowed, or null if the action is allowed.") + public String getReasonNotAllowed() { + return reasonNotAllowed; + } + + public void setReasonNotAllowed(final String reasonNotAllowed) { + this.reasonNotAllowed = reasonNotAllowed; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorConfigurationDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorConfigurationDTO.java new file mode 100644 index 000000000000..6ba11d535c01 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorConfigurationDTO.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.List; + +/** + * The configuration for a Connector. + */ +@XmlType(name = "connectorConfiguration") +public class ConnectorConfigurationDTO { + + private List configurationStepConfigurations; + + /** + * @return the configuration step configurations + */ + @Schema(description = "The list of configuration step configurations for this Connector.") + public List getConfigurationStepConfigurations() { + return configurationStepConfigurations; + } + + public void setConfigurationStepConfigurations(final List configurationStepConfigurations) { + this.configurationStepConfigurations = configurationStepConfigurations; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorDTO.java new file mode 100644 index 000000000000..ee891b10affe --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorDTO.java @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.Collection; +import java.util.List; + +/** + * Component representing a Connector instance. + */ +@XmlType(name = "connector") +public class ConnectorDTO extends ComponentDTO { + private String name; + private String type; + private BundleDTO bundle; + private String state; // RUNNING, STOPPED + private String managedProcessGroupId; + private ConnectorConfigurationDTO activeConfiguration; + private ConnectorConfigurationDTO workingConfiguration; + + private Collection validationErrors; + private String validationStatus; + private Boolean multipleVersionsAvailable; + private Boolean extensionMissing; + + private String configurationUrl; + private String detailsUrl; + private List availableActions; + + @Schema(description = "The name of the Connector.") + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + @Schema(description = "The fully qualified type of the Connector.") + public String getType() { + return type; + } + + public void setType(final String type) { + this.type = type; + } + + @Schema(description = "The details of the artifact that bundled this Connector type.") + public BundleDTO getBundle() { + return bundle; + } + + public void setBundle(final BundleDTO bundle) { + this.bundle = bundle; + } + + @Schema(description = "The state of the Connector.", allowableValues = {"RUNNING", "STOPPED"}) + public String getState() { + return state; + } + + public void setState(final String state) { + this.state = state; + } + + @Schema(description = "The identifier of the root Process Group managed by this Connector.") + public String getManagedProcessGroupId() { + return managedProcessGroupId; + } + + public void setManagedProcessGroupId(final String managedProcessGroupId) { + this.managedProcessGroupId = managedProcessGroupId; + } + + @Schema(description = "The active configuration of the Connector.") + public ConnectorConfigurationDTO getActiveConfiguration() { + return activeConfiguration; + } + + public void setActiveConfiguration(final ConnectorConfigurationDTO activeConfiguration) { + this.activeConfiguration = activeConfiguration; + } + + @Schema(description = "The working configuration of the Connector.") + public ConnectorConfigurationDTO getWorkingConfiguration() { + return workingConfiguration; + } + + public void setWorkingConfiguration(final ConnectorConfigurationDTO workingConfiguration) { + this.workingConfiguration = workingConfiguration; + } + + @Schema(description = "The validation errors for the connector.") + public Collection getValidationErrors() { + return validationErrors; + } + + public void setValidationErrors(final Collection validationErrors) { + this.validationErrors = validationErrors; + } + + @Schema(description = "Indicates whether the Connector is valid, invalid, or still in the process of validating") + public String getValidationStatus() { + return validationStatus; + } + + public void setValidationStatus(final String validationStatus) { + this.validationStatus = validationStatus; + } + + @Schema(description = "Whether multiple versions of this connector are available.") + public Boolean getMultipleVersionsAvailable() { + return multipleVersionsAvailable; + } + + public void setMultipleVersionsAvailable(final Boolean multipleVersionsAvailable) { + this.multipleVersionsAvailable = multipleVersionsAvailable; + } + + @Schema(description = "The URL for this connector's configuration/wizard custom UI, if applicable.") + public String getConfigurationUrl() { + return configurationUrl; + } + + public void setConfigurationUrl(final String configurationUrl) { + this.configurationUrl = configurationUrl; + } + + @Schema(description = "The URL for this connector's details custom UI, if applicable.") + public String getDetailsUrl() { + return detailsUrl; + } + + public void setDetailsUrl(final String detailsUrl) { + this.detailsUrl = detailsUrl; + } + + @Schema(description = "Whether the extension for this connector is missing.") + public Boolean getExtensionMissing() { + return extensionMissing; + } + + public void setExtensionMissing(final Boolean extensionMissing) { + this.extensionMissing = extensionMissing; + } + + @Schema(description = "The available actions that can be performed on this Connector.") + public List getAvailableActions() { + return availableActions; + } + + public void setAvailableActions(final List availableActions) { + this.availableActions = availableActions; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorPropertyDependencyDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorPropertyDependencyDTO.java new file mode 100644 index 000000000000..2604448a7a7c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorPropertyDependencyDTO.java @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.Set; + +/** + * A dependency that a connector property has on another property. + */ +@XmlType(name = "connectorPropertyDependency") +public class ConnectorPropertyDependencyDTO { + + private String propertyName; + private Set dependentValues; + + /** + * @return the name of the property that this property depends on + */ + @Schema(description = "The name of the property that this property depends on.") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(final String propertyName) { + this.propertyName = propertyName; + } + + /** + * @return the values of the dependent property that must be set for this property to be applicable + */ + @Schema(description = "The values of the dependent property that must be set for this property to be applicable.") + public Set getDependentValues() { + return dependentValues; + } + + public void setDependentValues(final Set dependentValues) { + this.dependentValues = dependentValues; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorPropertyDescriptorDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorPropertyDescriptorDTO.java new file mode 100644 index 000000000000..edc1e9c780ae --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorPropertyDescriptorDTO.java @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; +import org.apache.nifi.web.api.entity.AllowableValueEntity; + +import java.util.List; +import java.util.Set; + +/** + * A description of a connector property. + */ +@XmlType(name = "connectorPropertyDescriptor") +public class ConnectorPropertyDescriptorDTO { + + private String name; + private String description; + private String defaultValue; + private Boolean required; + private String type; + private List allowableValues; + private Boolean allowableValuesFetchable; + private Set dependencies; + + /** + * @return property name + */ + @Schema(description = "The name of the property.") + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + /** + * @return An explanation of the meaning of the given property + */ + @Schema(description = "The description of the property.") + public String getDescription() { + return description; + } + + public void setDescription(final String description) { + this.description = description; + } + + /** + * @return default value for this property + */ + @Schema(description = "The default value for the property.") + public String getDefaultValue() { + return defaultValue; + } + + public void setDefaultValue(final String defaultValue) { + this.defaultValue = defaultValue; + } + + /** + * @return whether this property is required + */ + @Schema(description = "Whether the property is required.") + public Boolean getRequired() { + return required; + } + + public void setRequired(final Boolean required) { + this.required = required; + } + + /** + * @return the property type (STRING, INTEGER, BOOLEAN, DOUBLE, STRING_LIST, etc.) + */ + @Schema(description = "The type of the property (STRING, INTEGER, BOOLEAN, DOUBLE, STRING_LIST).") + public String getType() { + return type; + } + + public void setType(final String type) { + this.type = type; + } + + /** + * @return set of allowable values for this property. If empty then the allowable values are not constrained + */ + @Schema(description = "Allowable values for the property. If empty then the allowed values are not constrained.") + public List getAllowableValues() { + return allowableValues; + } + + public void setAllowableValues(final List allowableValues) { + this.allowableValues = allowableValues; + } + + /** + * @return whether the allowable values can be fetched dynamically + */ + @Schema(description = "Whether the allowable values are dynamically fetchable based on other property values.") + public Boolean getAllowableValuesFetchable() { + return allowableValuesFetchable; + } + + public void setAllowableValuesFetchable(final Boolean allowableValuesFetchable) { + this.allowableValuesFetchable = allowableValuesFetchable; + } + + /** + * @return the dependencies this property has on other properties + */ + @Schema(description = "The dependencies this property has on other properties.") + public Set getDependencies() { + return dependencies; + } + + public void setDependencies(final Set dependencies) { + this.dependencies = dependencies; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorValueReferenceDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorValueReferenceDTO.java new file mode 100644 index 000000000000..ed79bdb3ac00 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ConnectorValueReferenceDTO.java @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.List; + +/** + * A reference to a connector property value, which includes the value and its type. + * The value can be a string literal, a reference to an asset, or a reference to a secret. + */ +@XmlType(name = "connectorValueReference") +public class ConnectorValueReferenceDTO { + + private String valueType; + private String value; + private String secretProviderId; + private String secretProviderName; + private String secretName; + private String fullyQualifiedSecretName; + private List assetReferences; + + /** + * @return the type of value (STRING_LITERAL, ASSET_REFERENCE, or SECRET_REFERENCE) + */ + @Schema(description = "The type of value (STRING_LITERAL, ASSET_REFERENCE, or SECRET_REFERENCE).") + public String getValueType() { + return valueType; + } + + public void setValueType(final String valueType) { + this.valueType = valueType; + } + + /** + * @return the string literal value when valueType is STRING_LITERAL + */ + @Schema(description = "The string literal value. Applicable when valueType is STRING_LITERAL.") + public String getValue() { + return value; + } + + public void setValue(final String value) { + this.value = value; + } + + /** + * @return the secret provider identifier when valueType is SECRET_REFERENCE + */ + @Schema(description = "The secret provider identifier. Applicable when valueType is SECRET_REFERENCE.") + public String getSecretProviderId() { + return secretProviderId; + } + + public void setSecretProviderId(final String secretProviderId) { + this.secretProviderId = secretProviderId; + } + + /** + * @return the secret provider name when valueType is SECRET_REFERENCE + */ + @Schema(description = "The secret provider name. Applicable when valueType is SECRET_REFERENCE.") + public String getSecretProviderName() { + return secretProviderName; + } + + public void setSecretProviderName(final String secretProviderName) { + this.secretProviderName = secretProviderName; + } + + /** + * @return the secret name when valueType is SECRET_REFERENCE + */ + @Schema(description = "The secret name. Applicable when valueType is SECRET_REFERENCE.") + public String getSecretName() { + return secretName; + } + + public void setSecretName(final String secretName) { + this.secretName = secretName; + } + + public void setFullyQualifiedSecretName(final String fullyQualifiedSecretName) { + this.fullyQualifiedSecretName = fullyQualifiedSecretName; + } + + @Schema(description = "The fully qualified secret name. Applicable when valueType is SECRET_REFERENCE.") + public String getFullyQualifiedSecretName() { + return fullyQualifiedSecretName; + } + + /** + * @return the asset identifiers when valueType is ASSET_REFERENCE + */ + @Schema(description = "The asset references. Applicable when valueType is ASSET_REFERENCE.") + public List getAssetReferences() { + return assetReferences; + } + + public void setAssetReferences(final List assetReferences) { + this.assetReferences = assetReferences; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/PropertyGroupConfigurationDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/PropertyGroupConfigurationDTO.java new file mode 100644 index 000000000000..a78f946e41fc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/PropertyGroupConfigurationDTO.java @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.Map; + +/** + * The configuration for a property group. + */ +@XmlType(name = "propertyGroupConfiguration") +public class PropertyGroupConfigurationDTO { + + private String propertyGroupName; + private String propertyGroupDescription; + private Map propertyDescriptors; + private Map propertyValues; + + /** + * @return the property group name + */ + @Schema(description = "The name of the property group.") + public String getPropertyGroupName() { + return propertyGroupName; + } + + public void setPropertyGroupName(final String propertyGroupName) { + this.propertyGroupName = propertyGroupName; + } + + /** + * @return the property group description + */ + @Schema(description = "The description of the property group.") + public String getPropertyGroupDescription() { + return propertyGroupDescription; + } + + public void setPropertyGroupDescription(final String propertyGroupDescription) { + this.propertyGroupDescription = propertyGroupDescription; + } + + /** + * @return the property descriptors keyed by property name + */ + @Schema(description = "The property descriptors for this property group, keyed by property name.") + public Map getPropertyDescriptors() { + return propertyDescriptors; + } + + public void setPropertyDescriptors(final Map propertyDescriptors) { + this.propertyDescriptors = propertyDescriptors; + } + + /** + * @return the property values + */ + @Schema(description = "The property values for this property group.") + public Map getPropertyValues() { + return propertyValues; + } + + public void setPropertyValues(final Map propertyValues) { + this.propertyValues = propertyValues; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/SecretDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/SecretDTO.java new file mode 100644 index 000000000000..dbbf0f36a0bd --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/SecretDTO.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +/** + * DTO representing a secret's metadata. Note: The actual secret value is never exposed via the REST API. + */ +@XmlType(name = "secret") +public class SecretDTO { + + private String providerId; + private String providerName; + private String groupName; + private String name; + private String fullyQualifiedName; + private String description; + + @Schema(description = "The identifier of the secret provider that manages this secret.") + public String getProviderId() { + return providerId; + } + + public void setProviderId(final String providerId) { + this.providerId = providerId; + } + + @Schema(description = "The name of the secret provider that manages this secret.") + public String getProviderName() { + return providerName; + } + + public void setProviderName(final String providerName) { + this.providerName = providerName; + } + + @Schema(description = "The name of the group this secret belongs to.") + public String getGroupName() { + return groupName; + } + + public void setGroupName(final String groupName) { + this.groupName = groupName; + } + + @Schema(description = "The name of the secret.") + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + @Schema(description = "The fully qualified name of the secret, including the group name.") + public String getFullyQualifiedName() { + return fullyQualifiedName; + } + + public void setFullyQualifiedName(final String fullyQualifiedName) { + this.fullyQualifiedName = fullyQualifiedName; + } + + @Schema(description = "A description of the secret.") + public String getDescription() { + return description; + } + + public void setDescription(final String description) { + this.description = description; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/VerifyConnectorConfigStepRequestDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/VerifyConnectorConfigStepRequestDTO.java new file mode 100644 index 000000000000..c42edd31c0f3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/VerifyConnectorConfigStepRequestDTO.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto; + +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.media.Schema.AccessMode; +import jakarta.xml.bind.annotation.XmlType; + +import java.util.List; + +@XmlType(name = "verifyConnectorConfigStepRequest") +public class VerifyConnectorConfigStepRequestDTO extends AsynchronousRequestDTO { + + private String connectorId; + private String configurationStepName; + private ConfigurationStepConfigurationDTO configurationStep; + private List results; + + @Schema(description = "The ID of the connector whose configuration step is being verified") + public String getConnectorId() { + return connectorId; + } + + public void setConnectorId(final String connectorId) { + this.connectorId = connectorId; + } + + @Schema(description = "The name of the configuration step being verified") + public String getConfigurationStepName() { + return configurationStepName; + } + + public void setConfigurationStepName(final String configurationStepName) { + this.configurationStepName = configurationStepName; + } + + @Schema(description = "The configuration step being verified") + public ConfigurationStepConfigurationDTO getConfigurationStep() { + return configurationStep; + } + + public void setConfigurationStep(final ConfigurationStepConfigurationDTO configurationStep) { + this.configurationStep = configurationStep; + } + + @Schema(description = "The results of the verification", accessMode = AccessMode.READ_ONLY) + public List getResults() { + return results; + } + + public void setResults(final List results) { + this.results = results; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/provenance/ProvenanceEventDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/provenance/ProvenanceEventDTO.java index 6c56c31258f6..88eeaa8a818f 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/provenance/ProvenanceEventDTO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/provenance/ProvenanceEventDTO.java @@ -49,6 +49,7 @@ public class ProvenanceEventDTO { private String clusterNodeAddress; // include when clustered private String groupId; + private String connectorId; private String componentId; private String componentType; private String componentName; @@ -217,6 +218,19 @@ public void setGroupId(String groupId) { this.groupId = groupId; } + /** + * @return id of the Connector that manages the component that generated this event, or {@code null} if the component is not managed by a Connector + */ + @Schema(description = "The id of the connector that manages the component that generated the event. If the component is not managed by a connector, this will not be set." + ) + public String getConnectorId() { + return connectorId; + } + + public void setConnectorId(String connectorId) { + this.connectorId = connectorId; + } + /** * @return id of the component that generated this event */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/ConnectorStatusDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/ConnectorStatusDTO.java new file mode 100644 index 000000000000..388c22dff540 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/ConnectorStatusDTO.java @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto.status; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; +import jakarta.xml.bind.annotation.adapters.XmlJavaTypeAdapter; +import org.apache.nifi.web.api.dto.util.TimeAdapter; + +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +/** + * DTO for serializing the status of a connector. + */ +@XmlType(name = "connectorStatus") +public class ConnectorStatusDTO implements Cloneable { + + private String id; + private String groupId; + private String name; + private String type; + private String runStatus; + private String validationStatus; + private Date statsLastRefreshed; + + private ConnectorStatusSnapshotDTO aggregateSnapshot; + private List nodeSnapshots; + + /** + * The id of the connector. + * + * @return The connector id + */ + @Schema(description = "The id of the connector.") + public String getId() { + return id; + } + + public void setId(final String id) { + this.id = id; + } + + /** + * The id of the group this connector belongs to. + * + * @return The group id + */ + @Schema(description = "The id of the group this connector belongs to.") + public String getGroupId() { + return groupId; + } + + public void setGroupId(final String groupId) { + this.groupId = groupId; + } + + /** + * The name of the connector. + * + * @return The connector name + */ + @Schema(description = "The name of the connector.") + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + /** + * The type of the connector. + * + * @return The connector type + */ + @Schema(description = "The type of the connector.") + public String getType() { + return type; + } + + public void setType(final String type) { + this.type = type; + } + + /** + * The run status of the connector. + * + * @return The run status + */ + @Schema(description = "The run status of the connector.") + public String getRunStatus() { + return runStatus; + } + + public void setRunStatus(final String runStatus) { + this.runStatus = runStatus; + } + + /** + * The validation status of the connector. + * + * @return The validation status + */ + @Schema(description = "The validation status of the connector.", + allowableValues = {"VALID", "INVALID", "VALIDATING"}) + public String getValidationStatus() { + return validationStatus; + } + + public void setValidationStatus(final String validationStatus) { + this.validationStatus = validationStatus; + } + + @XmlJavaTypeAdapter(TimeAdapter.class) + @Schema(description = "The timestamp of when the stats were last refreshed.", type = "string") + public Date getStatsLastRefreshed() { + return statsLastRefreshed; + } + + public void setStatsLastRefreshed(final Date statsLastRefreshed) { + this.statsLastRefreshed = statsLastRefreshed; + } + + @Schema(description = "A status snapshot that represents the aggregate stats of all nodes in the cluster. If the NiFi instance is " + + "a standalone instance, rather than a cluster, this represents the stats of the single instance.") + public ConnectorStatusSnapshotDTO getAggregateSnapshot() { + return aggregateSnapshot; + } + + public void setAggregateSnapshot(final ConnectorStatusSnapshotDTO aggregateSnapshot) { + this.aggregateSnapshot = aggregateSnapshot; + } + + @Schema(description = "A status snapshot for each node in the cluster. If the NiFi instance is a standalone instance, rather than " + + "a cluster, this may be null.") + public List getNodeSnapshots() { + return nodeSnapshots; + } + + public void setNodeSnapshots(final List nodeSnapshots) { + this.nodeSnapshots = nodeSnapshots; + } + + @Override + public ConnectorStatusDTO clone() { + final ConnectorStatusDTO other = new ConnectorStatusDTO(); + other.setId(getId()); + other.setGroupId(getGroupId()); + other.setName(getName()); + other.setType(getType()); + other.setRunStatus(getRunStatus()); + other.setValidationStatus(getValidationStatus()); + other.setStatsLastRefreshed(getStatsLastRefreshed()); + + if (getAggregateSnapshot() != null) { + other.setAggregateSnapshot(getAggregateSnapshot().clone()); + } + + final List snapshots = getNodeSnapshots(); + if (snapshots != null) { + final List snapshotClones = new ArrayList<>(snapshots.size()); + for (final NodeConnectorStatusSnapshotDTO snapshot : snapshots) { + snapshotClones.add(snapshot.clone()); + } + other.setNodeSnapshots(snapshotClones); + } + + return other; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/ConnectorStatusSnapshotDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/ConnectorStatusSnapshotDTO.java new file mode 100644 index 000000000000..23833dfb9733 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/ConnectorStatusSnapshotDTO.java @@ -0,0 +1,322 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto.status; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +/** + * DTO for serializing the status snapshot of a connector. + */ +@XmlType(name = "connectorStatusSnapshot") +public class ConnectorStatusSnapshotDTO implements Cloneable { + + private String id; + private String groupId; + private String name; + private String type; + private String runStatus; + + private Integer flowFilesSent = 0; + private Long bytesSent = 0L; + private Integer flowFilesReceived = 0; + private Long bytesReceived = 0L; + private Long bytesRead = 0L; + private Long bytesWritten = 0L; + private String sent; + private String received; + private String read; + private String written; + + private Integer flowFilesQueued = 0; + private Long bytesQueued = 0L; + private String queued; + private String queuedCount; + private String queuedSize; + + private Integer activeThreadCount = 0; + + private ProcessingPerformanceStatusDTO processingPerformanceStatus; + + private Boolean idle; + private Long idleDurationMillis; + private String idleDuration; + + @Schema(description = "The id of the connector.") + public String getId() { + return id; + } + + public void setId(final String id) { + this.id = id; + } + + @Schema(description = "The id of the parent process group to which the connector belongs.") + public String getGroupId() { + return groupId; + } + + public void setGroupId(final String groupId) { + this.groupId = groupId; + } + + @Schema(description = "The name of the connector.") + public String getName() { + return name; + } + + public void setName(final String name) { + this.name = name; + } + + @Schema(description = "The type of the connector.") + public String getType() { + return type; + } + + public void setType(final String type) { + this.type = type; + } + + @Schema(description = "The run status of the connector.") + public String getRunStatus() { + return runStatus; + } + + public void setRunStatus(final String runStatus) { + this.runStatus = runStatus; + } + + @Schema(description = "The number of FlowFiles sent by this connector's managed process group.") + public Integer getFlowFilesSent() { + return flowFilesSent; + } + + public void setFlowFilesSent(final Integer flowFilesSent) { + this.flowFilesSent = flowFilesSent; + } + + @Schema(description = "The number of bytes sent by this connector's managed process group.") + public Long getBytesSent() { + return bytesSent; + } + + public void setBytesSent(final Long bytesSent) { + this.bytesSent = bytesSent; + } + + @Schema(description = "The number of FlowFiles received by this connector's managed process group.") + public Integer getFlowFilesReceived() { + return flowFilesReceived; + } + + public void setFlowFilesReceived(final Integer flowFilesReceived) { + this.flowFilesReceived = flowFilesReceived; + } + + @Schema(description = "The number of bytes received by this connector's managed process group.") + public Long getBytesReceived() { + return bytesReceived; + } + + public void setBytesReceived(final Long bytesReceived) { + this.bytesReceived = bytesReceived; + } + + @Schema(description = "The number of bytes read by processors in this connector's managed process group.") + public Long getBytesRead() { + return bytesRead; + } + + public void setBytesRead(final Long bytesRead) { + this.bytesRead = bytesRead; + } + + @Schema(description = "The number of bytes written by processors in this connector's managed process group.") + public Long getBytesWritten() { + return bytesWritten; + } + + public void setBytesWritten(final Long bytesWritten) { + this.bytesWritten = bytesWritten; + } + + @Schema(description = "The count/size of data that has been sent by this connector, pretty-printed.") + public String getSent() { + return sent; + } + + public void setSent(final String sent) { + this.sent = sent; + } + + @Schema(description = "The count/size of data that has been received by this connector, pretty-printed.") + public String getReceived() { + return received; + } + + public void setReceived(final String received) { + this.received = received; + } + + @Schema(description = "The number of bytes read, pretty-printed.") + public String getRead() { + return read; + } + + public void setRead(final String read) { + this.read = read; + } + + @Schema(description = "The number of bytes written, pretty-printed.") + public String getWritten() { + return written; + } + + public void setWritten(final String written) { + this.written = written; + } + + @Schema(description = "The number of FlowFiles queued in this connector's managed process group.") + public Integer getFlowFilesQueued() { + return flowFilesQueued; + } + + public void setFlowFilesQueued(final Integer flowFilesQueued) { + this.flowFilesQueued = flowFilesQueued; + } + + @Schema(description = "The number of bytes queued in this connector's managed process group.") + public Long getBytesQueued() { + return bytesQueued; + } + + public void setBytesQueued(final Long bytesQueued) { + this.bytesQueued = bytesQueued; + } + + @Schema(description = "The count/size of queued data, pretty-printed.") + public String getQueued() { + return queued; + } + + public void setQueued(final String queued) { + this.queued = queued; + } + + @Schema(description = "The count of queued FlowFiles, pretty-printed.") + public String getQueuedCount() { + return queuedCount; + } + + public void setQueuedCount(final String queuedCount) { + this.queuedCount = queuedCount; + } + + @Schema(description = "The size of queued data, pretty-printed.") + public String getQueuedSize() { + return queuedSize; + } + + public void setQueuedSize(final String queuedSize) { + this.queuedSize = queuedSize; + } + + @Schema(description = "The number of active threads for the connector.") + public Integer getActiveThreadCount() { + return activeThreadCount; + } + + public void setActiveThreadCount(final Integer activeThreadCount) { + this.activeThreadCount = activeThreadCount; + } + + @Schema(description = "The processing performance status of the processors in this connector's managed process group.") + public ProcessingPerformanceStatusDTO getProcessingPerformanceStatus() { + return processingPerformanceStatus; + } + + public void setProcessingPerformanceStatus(final ProcessingPerformanceStatusDTO processingPerformanceStatus) { + this.processingPerformanceStatus = processingPerformanceStatus; + } + + @Schema(description = "Whether or not the connector is currently idle (no FlowFiles queued and no FlowFiles processed recently).") + public Boolean getIdle() { + return idle; + } + + public void setIdle(final Boolean idle) { + this.idle = idle; + } + + @Schema(description = "The number of milliseconds the connector has been idle, or null if the connector is not idle.") + public Long getIdleDurationMillis() { + return idleDurationMillis; + } + + public void setIdleDurationMillis(final Long idleDurationMillis) { + this.idleDurationMillis = idleDurationMillis; + } + + @Schema(description = "A human-readable representation of how long the connector has been idle, or null if the connector is not idle.") + public String getIdleDuration() { + return idleDuration; + } + + public void setIdleDuration(final String idleDuration) { + this.idleDuration = idleDuration; + } + + @Override + public ConnectorStatusSnapshotDTO clone() { + final ConnectorStatusSnapshotDTO other = new ConnectorStatusSnapshotDTO(); + other.setId(getId()); + other.setGroupId(getGroupId()); + other.setName(getName()); + other.setType(getType()); + other.setRunStatus(getRunStatus()); + + other.setFlowFilesSent(getFlowFilesSent()); + other.setBytesSent(getBytesSent()); + other.setFlowFilesReceived(getFlowFilesReceived()); + other.setBytesReceived(getBytesReceived()); + other.setBytesRead(getBytesRead()); + other.setBytesWritten(getBytesWritten()); + other.setSent(getSent()); + other.setReceived(getReceived()); + other.setRead(getRead()); + other.setWritten(getWritten()); + + other.setFlowFilesQueued(getFlowFilesQueued()); + other.setBytesQueued(getBytesQueued()); + other.setQueued(getQueued()); + other.setQueuedCount(getQueuedCount()); + other.setQueuedSize(getQueuedSize()); + + other.setActiveThreadCount(getActiveThreadCount()); + + if (getProcessingPerformanceStatus() != null) { + other.setProcessingPerformanceStatus(getProcessingPerformanceStatus().clone()); + } + + other.setIdle(getIdle()); + other.setIdleDurationMillis(getIdleDurationMillis()); + other.setIdleDuration(getIdleDuration()); + + return other; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeConnectorStatusSnapshotDTO.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeConnectorStatusSnapshotDTO.java new file mode 100644 index 000000000000..de9cc43a6c83 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeConnectorStatusSnapshotDTO.java @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.dto.status; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +/** + * DTO for serializing the connector status snapshot for a particular node. + */ +@XmlType(name = "nodeConnectorStatusSnapshot") +public class NodeConnectorStatusSnapshotDTO implements Cloneable { + + private String nodeId; + private String address; + private Integer apiPort; + + private ConnectorStatusSnapshotDTO statusSnapshot; + + @Schema(description = "The unique ID that identifies the node") + public String getNodeId() { + return nodeId; + } + + public void setNodeId(final String nodeId) { + this.nodeId = nodeId; + } + + @Schema(description = "The API address of the node") + public String getAddress() { + return address; + } + + public void setAddress(final String address) { + this.address = address; + } + + @Schema(description = "The API port used to communicate with the node") + public Integer getApiPort() { + return apiPort; + } + + public void setApiPort(final Integer apiPort) { + this.apiPort = apiPort; + } + + @Schema(description = "The connector status snapshot from the node.") + public ConnectorStatusSnapshotDTO getStatusSnapshot() { + return statusSnapshot; + } + + public void setStatusSnapshot(final ConnectorStatusSnapshotDTO statusSnapshot) { + this.statusSnapshot = statusSnapshot; + } + + @Override + public NodeConnectorStatusSnapshotDTO clone() { + final NodeConnectorStatusSnapshotDTO other = new NodeConnectorStatusSnapshotDTO(); + other.setNodeId(getNodeId()); + other.setAddress(getAddress()); + other.setApiPort(getApiPort()); + other.setStatusSnapshot(getStatusSnapshot().clone()); + return other; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConfigurationStepEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConfigurationStepEntity.java new file mode 100644 index 000000000000..689b1ddf96fc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConfigurationStepEntity.java @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.RevisionDTO; + +/** + * A serialized representation of this class can be placed in the entity body of a request or response to or from the API. + * This particular entity holds a reference to a ConfigurationStepConfigurationDTO. + * Note that this entity does not extend ComponentEntity because a configuration step is not itself an updatable component. + * Instead, it is updated through the parent Connector, so the client must provide the parent connector ID and revision. + * Permissions are not included on this entity since they are managed at the parent Connector level. + */ +@XmlRootElement(name = "configurationStepEntity") +public class ConfigurationStepEntity extends Entity { + + private ConfigurationStepConfigurationDTO configurationStep; + private String parentConnectorId; + private RevisionDTO parentConnectorRevision; + private Boolean disconnectedNodeAcknowledged; + + /** + * @return the configuration step configuration + */ + @Schema(description = "The configuration step configuration.") + public ConfigurationStepConfigurationDTO getConfigurationStep() { + return configurationStep; + } + + public void setConfigurationStep(final ConfigurationStepConfigurationDTO configurationStep) { + this.configurationStep = configurationStep; + } + + @Schema(description = "The id of the parent connector.") + public String getParentConnectorId() { + return parentConnectorId; + } + + public void setParentConnectorId(final String parentConnectorId) { + this.parentConnectorId = parentConnectorId; + } + + @Schema(description = "The revision of the parent connector that this configuration step belongs to.") + public RevisionDTO getParentConnectorRevision() { + return parentConnectorRevision; + } + + public void setParentConnectorRevision(final RevisionDTO parentConnectorRevision) { + this.parentConnectorRevision = parentConnectorRevision; + } + + @Schema(description = "Acknowledges that this node is disconnected to allow for mutable requests to proceed.") + public Boolean isDisconnectedNodeAcknowledged() { + return disconnectedNodeAcknowledged; + } + + public void setDisconnectedNodeAcknowledged(final Boolean disconnectedNodeAcknowledged) { + this.disconnectedNodeAcknowledged = disconnectedNodeAcknowledged; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConfigurationStepNamesEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConfigurationStepNamesEntity.java new file mode 100644 index 000000000000..0dc2da053655 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConfigurationStepNamesEntity.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; + +import java.util.List; + +/** + * A serialized representation of this class can be placed in the entity body of a request or response to or from the API. + * This particular entity holds a reference to a list of configuration step names. + */ +@XmlRootElement(name = "configurationStepNamesEntity") +public class ConfigurationStepNamesEntity extends Entity { + + private List configurationStepNames; + + /** + * @return the configuration step names + */ + @Schema(description = "The list of configuration step names for a connector.") + public List getConfigurationStepNames() { + return configurationStepNames; + } + + public void setConfigurationStepNames(final List configurationStepNames) { + this.configurationStepNames = configurationStepNames; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorEntity.java new file mode 100644 index 000000000000..0cf02aaa00d3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorEntity.java @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; +import jakarta.xml.bind.annotation.XmlType; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.PermissionsDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; + +@XmlRootElement(name = "connectorEntity") +@XmlType(name = "connectorEntity") +public class ConnectorEntity extends ComponentEntity implements Permissible, OperationPermissible { + private ConnectorDTO component; + private ConnectorStatusDTO status; + private PermissionsDTO operatePermissions; + + @Override + @Schema(description = "The Connector DTO") + public ConnectorDTO getComponent() { + return component; + } + + @Override + public void setComponent(final ConnectorDTO component) { + this.component = component; + } + + @Schema(description = "The permissions for this component operations.") + @Override + public PermissionsDTO getOperatePermissions() { + return operatePermissions; + } + + @Override + public void setOperatePermissions(final PermissionsDTO operatePermissions) { + this.operatePermissions = operatePermissions; + } + + @Schema(description = "The status of the connector.") + public ConnectorStatusDTO getStatus() { + return status; + } + + public void setStatus(final ConnectorStatusDTO status) { + this.status = status; + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorPropertyAllowableValuesEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorPropertyAllowableValuesEntity.java new file mode 100644 index 000000000000..a2be3a0483cf --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorPropertyAllowableValuesEntity.java @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; + +import java.util.List; + +/** + * Entity for holding allowable values for a connector property. + */ +@XmlRootElement(name = "connectorPropertyAllowableValuesEntity") +public class ConnectorPropertyAllowableValuesEntity extends Entity { + + private String configurationStepName; + private String propertyGroupName; + private String propertyName; + private List allowableValues; + + @Schema(description = "The name of the configuration step.") + public String getConfigurationStepName() { + return configurationStepName; + } + + public void setConfigurationStepName(final String configurationStepName) { + this.configurationStepName = configurationStepName; + } + + @Schema(description = "The name of the property group.") + public String getPropertyGroupName() { + return propertyGroupName; + } + + public void setPropertyGroupName(final String propertyGroupName) { + this.propertyGroupName = propertyGroupName; + } + + @Schema(description = "The name of the property.") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(final String propertyName) { + this.propertyName = propertyName; + } + + @Schema(description = "The allowable values for the property.") + public List getAllowableValues() { + return allowableValues; + } + + public void setAllowableValues(final List allowableValues) { + this.allowableValues = allowableValues; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorRunStatusEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorRunStatusEntity.java new file mode 100644 index 000000000000..cd08a7fa7603 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorRunStatusEntity.java @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlType; + +@XmlType(name = "connectorRunStatus") +public class ConnectorRunStatusEntity extends ComponentRunStatusEntity { + private static final String[] SUPPORTED_STATE = {"RUNNING", "STOPPED", "DISABLED"}; + + @Override + protected String[] getSupportedState() { + return SUPPORTED_STATE; + } + + @Schema(description = "The run status of the Connector.", allowableValues = {"RUNNING", "STOPPED", "DISABLED"}) + @Override + public String getState() { + return super.getState(); + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorTypesEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorTypesEntity.java new file mode 100644 index 000000000000..3192d3cab6dc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorTypesEntity.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import jakarta.xml.bind.annotation.XmlRootElement; +import org.apache.nifi.web.api.dto.DocumentedTypeDTO; + +import java.util.Set; + +/** + * A serialized representation of this class can be placed in the entity body of a response to the API. This particular entity holds a reference to a list of connector types. + */ +@XmlRootElement(name = "connectorTypesEntity") +public class ConnectorTypesEntity extends Entity { + + private Set connectorTypes; + + /** + * @return list of connector types that are being serialized + */ + public Set getConnectorTypes() { + return connectorTypes; + } + + public void setConnectorTypes(final Set connectorTypes) { + this.connectorTypes = connectorTypes; + } + +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorsEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorsEntity.java new file mode 100644 index 000000000000..46219d6b1261 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ConnectorsEntity.java @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; +import jakarta.xml.bind.annotation.XmlType; +import jakarta.xml.bind.annotation.adapters.XmlJavaTypeAdapter; +import org.apache.nifi.web.api.dto.util.TimeAdapter; + +import java.util.Date; +import java.util.Set; + +@XmlRootElement(name = "connectorsEntity") +@XmlType(name = "connectorsEntity") +public class ConnectorsEntity extends Entity { + private Date currentTime; + private Set connectors; + + @XmlJavaTypeAdapter(TimeAdapter.class) + @Schema(description = "The current time on the system.", + type = "string" + ) + public Date getCurrentTime() { + return currentTime; + } + + public void setCurrentTime(final Date currentTime) { + this.currentTime = currentTime; + } + + public Set getConnectors() { + return connectors; + } + + public void setConnectors(final Set connectors) { + this.connectors = connectors; + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/CurrentUserEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/CurrentUserEntity.java index 0756eaf66dd2..8bc5b65878a3 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/CurrentUserEntity.java +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/CurrentUserEntity.java @@ -40,6 +40,7 @@ public class CurrentUserEntity extends Entity { private PermissionsDTO policiesPermissions; private PermissionsDTO systemPermissions; private PermissionsDTO parameterContextPermissions; + private PermissionsDTO connectorsPermissions; private PermissionsDTO restrictedComponentsPermissions; private Set componentRestrictionPermissions; @@ -165,6 +166,18 @@ public void setParameterContextPermissions(PermissionsDTO parameterContextPermis this.parameterContextPermissions = parameterContextPermissions; } + /** + * @return permissions for accessing connectors + */ + @Schema(description = "Permissions for accessing connectors.") + public PermissionsDTO getConnectorsPermissions() { + return connectorsPermissions; + } + + public void setConnectorsPermissions(PermissionsDTO connectorsPermissions) { + this.connectorsPermissions = connectorsPermissions; + } + /** * @return permissions for accessing the restricted components */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/NarDetailsEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/NarDetailsEntity.java index 799677beeea1..35c4735e47b8 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/NarDetailsEntity.java +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/NarDetailsEntity.java @@ -36,6 +36,7 @@ public class NarDetailsEntity extends Entity { private Set parameterProviderTypes; private Set flowRegistryClientTypes; private Set flowAnalysisRuleTypes; + private Set connectorTypes; @Schema(description = "The NAR summary") public NarSummaryDTO getNarSummary() { @@ -108,4 +109,13 @@ public Set getFlowAnalysisRuleTypes() { public void setFlowAnalysisRuleTypes(final Set flowAnalysisRuleTypes) { this.flowAnalysisRuleTypes = flowAnalysisRuleTypes; } + + @Schema(description = "The Connector types contained in the NAR") + public Set getConnectorTypes() { + return connectorTypes; + } + + public void setConnectorTypes(final Set connectorTypes) { + this.connectorTypes = connectorTypes; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/SecretsEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/SecretsEntity.java new file mode 100644 index 000000000000..0706dde05c97 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/SecretsEntity.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; +import org.apache.nifi.web.api.dto.SecretDTO; + +import java.util.List; + +/** + * A serialized representation of this class can be placed in the entity body of a response to the API. + * This particular entity holds a list of secrets. + */ +@XmlRootElement(name = "secretsEntity") +public class SecretsEntity extends Entity { + + private List secrets; + + @Schema(description = "The list of secrets available from all secret providers.") + public List getSecrets() { + return secrets; + } + + public void setSecrets(final List secrets) { + this.secrets = secrets; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/StepDocumentationEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/StepDocumentationEntity.java new file mode 100644 index 000000000000..e01da34a1dcd --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/StepDocumentationEntity.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import jakarta.xml.bind.annotation.XmlRootElement; + +@XmlRootElement(name = "stepDocumentationEntity") +public class StepDocumentationEntity extends Entity { + + private String stepDocumentation; + + public String getStepDocumentation() { + return stepDocumentation; + } + + public void setStepDocumentation(final String stepDocumentation) { + this.stepDocumentation = stepDocumentation; + } + +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/VerifyConnectorConfigStepRequestEntity.java b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/VerifyConnectorConfigStepRequestEntity.java new file mode 100644 index 000000000000..cacfe93c3306 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/VerifyConnectorConfigStepRequestEntity.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.entity; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlRootElement; +import org.apache.nifi.web.api.dto.VerifyConnectorConfigStepRequestDTO; + +@XmlRootElement(name = "verifyConnectorConfigStepRequestEntity") +public class VerifyConnectorConfigStepRequestEntity extends Entity { + + private VerifyConnectorConfigStepRequestDTO request; + + @Schema(description = "The verification request") + public VerifyConnectorConfigStepRequestDTO getRequest() { + return request; + } + + public void setRequest(final VerifyConnectorConfigStepRequestDTO request) { + this.request = request; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/ReconstitutedSerializedRepositoryRecord.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/ReconstitutedSerializedRepositoryRecord.java index bda74586d8b6..55fc92024ada 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/ReconstitutedSerializedRepositoryRecord.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/ReconstitutedSerializedRepositoryRecord.java @@ -24,12 +24,14 @@ public class ReconstitutedSerializedRepositoryRecord implements SerializedReposi private final RepositoryRecordType type; private final FlowFileRecord flowFile; private final String swapLocation; + private final String originalSwapLocation; private ReconstitutedSerializedRepositoryRecord(final Builder builder) { this.queueIdentifier = builder.queueIdentifier; this.type = builder.type; this.flowFile = builder.flowFile; this.swapLocation = builder.swapLocation; + this.originalSwapLocation = builder.originalSwapLocation; } @Override @@ -72,6 +74,11 @@ public String getSwapLocation() { return swapLocation; } + @Override + public String getOriginalSwapLocation() { + return originalSwapLocation; + } + @Override public String toString() { return "ReconstitutedSerializedRepositoryRecord[recordType=" + type + ", queueId=" + queueIdentifier + ", flowFileUuid=" + flowFile.getAttribute(CoreAttributes.UUID.key()) @@ -83,6 +90,7 @@ public static class Builder { private RepositoryRecordType type; private FlowFileRecord flowFile; private String swapLocation; + private String originalSwapLocation; public Builder queueIdentifier(final String queueIdentifier) { this.queueIdentifier = queueIdentifier; @@ -104,6 +112,11 @@ public Builder swapLocation(final String swapLocation) { return this; } + public Builder originalSwapLocation(final String originalSwapLocation) { + this.originalSwapLocation = originalSwapLocation; + return this; + } + public ReconstitutedSerializedRepositoryRecord build() { return new ReconstitutedSerializedRepositoryRecord(this); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordSerde.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordSerde.java index b3faad6c2811..dbee510fa7b0 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordSerde.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/RepositoryRecordSerde.java @@ -24,7 +24,11 @@ public abstract class RepositoryRecordSerde implements SerDe UpdateType.UPDATE; case SWAP_OUT -> UpdateType.SWAP_OUT; case SWAP_IN -> UpdateType.SWAP_IN; + case SWAP_FILE_DELETED -> UpdateType.SWAP_FILE_DELETED; + case SWAP_FILE_RENAMED -> UpdateType.SWAP_FILE_RENAMED; default -> null; }; } @@ -43,4 +49,9 @@ public UpdateType getUpdateType(final SerializedRepositoryRecord record) { public String getLocation(final SerializedRepositoryRecord record) { return record.getSwapLocation(); } + + @Override + public String getOriginalLocation(final SerializedRepositoryRecord record) { + return record.getOriginalSwapLocation(); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerde.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerde.java index 767fe5d0d45f..b2ea001708b9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerde.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerde.java @@ -44,9 +44,9 @@ import java.util.Map; public class SchemaRepositoryRecordSerde extends RepositoryRecordSerde implements SerDe { - private static final int MAX_ENCODING_VERSION = 2; + private static final int MAX_ENCODING_VERSION = 3; - private final RecordSchema writeSchema = RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V2; + private final RecordSchema writeSchema = RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V3; private final RecordSchema contentClaimSchema = ContentClaimSchema.CONTENT_CLAIM_SCHEMA_V1; private final ResourceClaimManager resourceClaimManager; @@ -76,11 +76,13 @@ public void serializeRecord(final SerializedRepositoryRecord record, final DataO case CONTENTMISSING, DELETE -> RepositoryRecordSchema.DELETE_SCHEMA_V2; case SWAP_IN -> RepositoryRecordSchema.SWAP_IN_SCHEMA_V2; case SWAP_OUT -> RepositoryRecordSchema.SWAP_OUT_SCHEMA_V2; + case SWAP_FILE_DELETED -> RepositoryRecordSchema.SWAP_FILE_DELETED_SCHEMA_V3; + case SWAP_FILE_RENAMED -> RepositoryRecordSchema.SWAP_FILE_RENAMED_SCHEMA_V3; default -> throw new IllegalArgumentException("Received Repository Record with unknown Update Type: " + record.getType()); // won't happen. }; - serializeRecord(record, out, schema, RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V2); + serializeRecord(record, out, schema, RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V3); } protected void serializeRecord(final SerializedRepositoryRecord record, final DataOutputStream out, RecordSchema schema, RecordSchema repositoryRecordSchema) throws IOException { @@ -163,6 +165,8 @@ private SerializedRepositoryRecord createRepositoryRecord(final Record updateRec case SWAP_IN -> swapInRecord(record); case SWAP_OUT -> swapOutRecord(record); case UPDATE -> updateRecord(record); + case SWAP_FILE_DELETED -> swapFileDeletedRecord(record); + case SWAP_FILE_RENAMED -> swapFileRenamedRecord(record); default -> throw new IOException("Found unrecognized Update Type '" + actionType + "'"); }; @@ -254,6 +258,30 @@ private SerializedRepositoryRecord swapOutRecord(final Record record) { .build(); } + private SerializedRepositoryRecord swapFileDeletedRecord(final Record record) { + final String queueId = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE)); + final String swapLocation = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + + return new ReconstitutedSerializedRepositoryRecord.Builder() + .type(RepositoryRecordType.SWAP_FILE_DELETED) + .swapLocation(swapLocation) + .queueIdentifier(queueId) + .build(); + } + + private SerializedRepositoryRecord swapFileRenamedRecord(final Record record) { + final String queueId = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE)); + final String originalSwapLocation = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.ORIGINAL_SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + final String swapLocation = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + + return new ReconstitutedSerializedRepositoryRecord.Builder() + .type(RepositoryRecordType.SWAP_FILE_RENAMED) + .originalSwapLocation(originalSwapLocation) + .swapLocation(swapLocation) + .queueIdentifier(queueId) + .build(); + } + @Override public int getVersion() { return MAX_ENCODING_VERSION; diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SerializedRepositoryRecord.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SerializedRepositoryRecord.java index 51eb034cd92f..5d87852fec8b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SerializedRepositoryRecord.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/SerializedRepositoryRecord.java @@ -45,6 +45,14 @@ public interface SerializedRepositoryRecord { */ String getSwapLocation(); + /** + * @return For SWAP_FILE_RENAMED records, provides the original swap location before the rename. + * For other record types, returns null. + */ + default String getOriginalSwapLocation() { + return null; + } + /** * @return FlowFile being encapsulated by this record */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecordSerdeFactory.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecordSerdeFactory.java index d466ffa8d0ac..effcc57ef396 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecordSerdeFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecordSerdeFactory.java @@ -55,7 +55,11 @@ public SerDe createSerDe(final String encodingName) @Override public Long getRecordIdentifier(final SerializedRepositoryRecord record) { - return record.getFlowFileRecord().getId(); + final FlowFileRecord flowFileRecord = record.getFlowFileRecord(); + if (flowFileRecord == null) { + return null; + } + return flowFileRecord.getId(); } @Override @@ -66,6 +70,8 @@ public UpdateType getUpdateType(final SerializedRepositoryRecord record) { case UPDATE -> UpdateType.UPDATE; case SWAP_OUT -> UpdateType.SWAP_OUT; case SWAP_IN -> UpdateType.SWAP_IN; + case SWAP_FILE_DELETED -> UpdateType.SWAP_FILE_DELETED; + case SWAP_FILE_RENAMED -> UpdateType.SWAP_FILE_RENAMED; default -> null; }; } @@ -75,4 +81,9 @@ public String getLocation(final SerializedRepositoryRecord record) { return record.getSwapLocation(); } + @Override + public String getOriginalLocation(final SerializedRepositoryRecord record) { + return record.getOriginalSwapLocation(); + } + } diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordFieldMap.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordFieldMap.java index 83c4b0791151..221b6860e1ab 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordFieldMap.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordFieldMap.java @@ -39,15 +39,16 @@ public RepositoryRecordFieldMap(final SerializedRepositoryRecord record, final R public Object getFieldValue(final String fieldName) { return switch (fieldName) { case RepositoryRecordSchema.ACTION_TYPE -> record.getType().name(); - case RepositoryRecordSchema.RECORD_ID -> record.getFlowFileRecord().getId(); + case RepositoryRecordSchema.RECORD_ID -> flowFile == null ? null : flowFile.getId(); case RepositoryRecordSchema.SWAP_LOCATION -> record.getSwapLocation(); - case FlowFileSchema.ATTRIBUTES -> flowFile.getAttributes(); - case FlowFileSchema.ENTRY_DATE -> flowFile.getEntryDate(); - case FlowFileSchema.FLOWFILE_SIZE -> flowFile.getSize(); - case FlowFileSchema.LINEAGE_START_DATE -> flowFile.getLineageStartDate(); - case FlowFileSchema.LINEAGE_START_INDEX -> flowFile.getLineageStartIndex(); - case FlowFileSchema.QUEUE_DATE -> flowFile.getLastQueueDate(); - case FlowFileSchema.QUEUE_DATE_INDEX -> flowFile.getQueueDateIndex(); + case RepositoryRecordSchema.ORIGINAL_SWAP_LOCATION -> record.getOriginalSwapLocation(); + case FlowFileSchema.ATTRIBUTES -> flowFile == null ? null : flowFile.getAttributes(); + case FlowFileSchema.ENTRY_DATE -> flowFile == null ? null : flowFile.getEntryDate(); + case FlowFileSchema.FLOWFILE_SIZE -> flowFile == null ? null : flowFile.getSize(); + case FlowFileSchema.LINEAGE_START_DATE -> flowFile == null ? null : flowFile.getLineageStartDate(); + case FlowFileSchema.LINEAGE_START_INDEX -> flowFile == null ? null : flowFile.getLineageStartIndex(); + case FlowFileSchema.QUEUE_DATE -> flowFile == null ? null : flowFile.getLastQueueDate(); + case FlowFileSchema.QUEUE_DATE_INDEX -> flowFile == null ? null : flowFile.getQueueDateIndex(); case FlowFileSchema.CONTENT_CLAIM -> { final ContentClaimFieldMap contentClaimFieldMap = record.getContentClaim() == null ? null : new ContentClaimFieldMap(record.getContentClaim(), record.getClaimOffset(), contentClaimSchema); diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordSchema.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordSchema.java index db77c8baf487..935a3046251b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordSchema.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordSchema.java @@ -32,18 +32,22 @@ public class RepositoryRecordSchema { public static final String REPOSITORY_RECORD_UPDATE_V1 = "Repository Record Update"; // top level field name public static final String REPOSITORY_RECORD_UPDATE_V2 = "Repository Record Update"; // top level field name + public static final String REPOSITORY_RECORD_UPDATE_V3 = "Repository Record Update"; // top level field name // repository record fields public static final String ACTION_TYPE = "Action"; public static final String RECORD_ID = "Record ID"; public static final String QUEUE_IDENTIFIER = "Queue Identifier"; public static final String SWAP_LOCATION = "Swap Location"; + public static final String ORIGINAL_SWAP_LOCATION = "Original Swap Location"; // Update types public static final String CREATE_OR_UPDATE_ACTION = "Create or Update"; public static final String DELETE_ACTION = "Delete"; public static final String SWAP_IN_ACTION = "Swap In"; public static final String SWAP_OUT_ACTION = "Swap Out"; + public static final String SWAP_FILE_DELETED_ACTION = "Swap File Deleted"; + public static final String SWAP_FILE_RENAMED_ACTION = "Swap File Renamed"; public static final RecordSchema REPOSITORY_RECORD_SCHEMA_V1; public static final RecordSchema CREATE_OR_UPDATE_SCHEMA_V1; @@ -57,6 +61,10 @@ public class RepositoryRecordSchema { public static final RecordSchema SWAP_IN_SCHEMA_V2; public static final RecordSchema SWAP_OUT_SCHEMA_V2; + public static final RecordSchema REPOSITORY_RECORD_SCHEMA_V3; + public static final RecordSchema SWAP_FILE_DELETED_SCHEMA_V3; + public static final RecordSchema SWAP_FILE_RENAMED_SCHEMA_V3; + public static final RecordField ACTION_TYPE_FIELD = new SimpleRecordField(ACTION_TYPE, FieldType.STRING, Repetition.EXACTLY_ONE); public static final RecordField RECORD_ID_FIELD = new SimpleRecordField(RECORD_ID, FieldType.LONG, Repetition.EXACTLY_ONE); @@ -132,7 +140,62 @@ public class RepositoryRecordSchema { SWAP_IN_SCHEMA_V2 = new RecordSchema(swapInFields); // Union Field that creates the top-level field type - final UnionRecordField repoUpdateField = new UnionRecordField(REPOSITORY_RECORD_UPDATE_V2, Repetition.EXACTLY_ONE, createOrUpdate, delete, swapOut, swapIn); + final UnionRecordField repoUpdateField = new UnionRecordField(REPOSITORY_RECORD_UPDATE_V2, Repetition.EXACTLY_ONE, + createOrUpdate, delete, swapOut, swapIn); REPOSITORY_RECORD_SCHEMA_V2 = new RecordSchema(Collections.singletonList(repoUpdateField)); } + + static { + // V3 reuses V2 schemas for Create/Update, Delete, Swap Out, Swap In + // and adds new schemas for Swap File Deleted and Swap File Renamed + + // Fields for "Create" or "Update" records (same as V2) + final List createOrUpdateFields = new ArrayList<>(); + createOrUpdateFields.add(ACTION_TYPE_FIELD); + createOrUpdateFields.addAll(FlowFileSchema.FLOWFILE_SCHEMA_V2.getFields()); + createOrUpdateFields.add(new SimpleRecordField(QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE)); + createOrUpdateFields.add(new SimpleRecordField(SWAP_LOCATION, FieldType.STRING, Repetition.ZERO_OR_ONE)); + final ComplexRecordField createOrUpdate = new ComplexRecordField(CREATE_OR_UPDATE_ACTION, Repetition.EXACTLY_ONE, createOrUpdateFields); + + // Fields for "Delete" records (same as V2) + final List deleteFields = new ArrayList<>(); + deleteFields.add(ACTION_TYPE_FIELD); + deleteFields.add(RECORD_ID_FIELD); + final ComplexRecordField delete = new ComplexRecordField(DELETE_ACTION, Repetition.EXACTLY_ONE, deleteFields); + + // Fields for "Swap Out" records (same as V2) + final List swapOutFields = new ArrayList<>(); + swapOutFields.add(ACTION_TYPE_FIELD); + swapOutFields.add(RECORD_ID_FIELD); + swapOutFields.add(new SimpleRecordField(QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE)); + swapOutFields.add(new SimpleRecordField(SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + final ComplexRecordField swapOut = new ComplexRecordField(SWAP_OUT_ACTION, Repetition.EXACTLY_ONE, swapOutFields); + + // Fields for "Swap In" records (same as V2) + final List swapInFields = new ArrayList<>(createOrUpdateFields); + swapInFields.add(new SimpleRecordField(SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + final ComplexRecordField swapIn = new ComplexRecordField(SWAP_IN_ACTION, Repetition.EXACTLY_ONE, swapInFields); + + // Fields for "Swap File Deleted" records (new in V3) + final List swapFileDeletedFields = new ArrayList<>(); + swapFileDeletedFields.add(ACTION_TYPE_FIELD); + swapFileDeletedFields.add(new SimpleRecordField(QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE)); + swapFileDeletedFields.add(new SimpleRecordField(SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + final ComplexRecordField swapFileDeleted = new ComplexRecordField(SWAP_FILE_DELETED_ACTION, Repetition.EXACTLY_ONE, swapFileDeletedFields); + SWAP_FILE_DELETED_SCHEMA_V3 = new RecordSchema(swapFileDeletedFields); + + // Fields for "Swap File Renamed" records (new in V3) + final List swapFileRenamedFields = new ArrayList<>(); + swapFileRenamedFields.add(ACTION_TYPE_FIELD); + swapFileRenamedFields.add(new SimpleRecordField(QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE)); + swapFileRenamedFields.add(new SimpleRecordField(ORIGINAL_SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + swapFileRenamedFields.add(new SimpleRecordField(SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE)); + final ComplexRecordField swapFileRenamed = new ComplexRecordField(SWAP_FILE_RENAMED_ACTION, Repetition.EXACTLY_ONE, swapFileRenamedFields); + SWAP_FILE_RENAMED_SCHEMA_V3 = new RecordSchema(swapFileRenamedFields); + + // Union Field that creates the top-level field type (V3 includes all V2 types plus new swap file types) + final UnionRecordField repoUpdateField = new UnionRecordField(REPOSITORY_RECORD_UPDATE_V3, Repetition.EXACTLY_ONE, + createOrUpdate, delete, swapOut, swapIn, swapFileDeleted, swapFileRenamed); + REPOSITORY_RECORD_SCHEMA_V3 = new RecordSchema(Collections.singletonList(repoUpdateField)); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordUpdate.java b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordUpdate.java index d93b23d73e97..5812561f45c8 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordUpdate.java +++ b/nifi-framework-bundle/nifi-framework/nifi-flowfile-repo-serialization/src/main/java/org/apache/nifi/controller/repository/schema/RepositoryRecordUpdate.java @@ -21,7 +21,6 @@ import org.apache.nifi.repository.schema.NamedValue; import org.apache.nifi.repository.schema.Record; import org.apache.nifi.repository.schema.RecordSchema; -import org.wali.UpdateType; public class RepositoryRecordUpdate implements Record { private final RecordSchema schema; @@ -40,17 +39,17 @@ public RecordSchema getSchema() { @Override public Object getFieldValue(final String fieldName) { if (RepositoryRecordSchema.REPOSITORY_RECORD_UPDATE_V2.equals(fieldName)) { - String actionType = (String) fieldMap.getFieldValue(RepositoryRecordSchema.ACTION_TYPE); - if (RepositoryRecordType.CONTENTMISSING.name().equals(actionType)) { - actionType = RepositoryRecordType.DELETE.name(); - } - final UpdateType updateType = UpdateType.valueOf(actionType); + final String actionType = (String) fieldMap.getFieldValue(RepositoryRecordSchema.ACTION_TYPE); + final RepositoryRecordType recordType = RepositoryRecordType.valueOf(actionType); - final String actionName = switch (updateType) { + final String actionName = switch (recordType) { case CREATE, UPDATE -> RepositoryRecordSchema.CREATE_OR_UPDATE_ACTION; - case DELETE -> RepositoryRecordSchema.DELETE_ACTION; + case DELETE, CONTENTMISSING -> RepositoryRecordSchema.DELETE_ACTION; case SWAP_IN -> RepositoryRecordSchema.SWAP_IN_ACTION; case SWAP_OUT -> RepositoryRecordSchema.SWAP_OUT_ACTION; + case SWAP_FILE_DELETED -> RepositoryRecordSchema.SWAP_FILE_DELETED_ACTION; + case SWAP_FILE_RENAMED -> RepositoryRecordSchema.SWAP_FILE_RENAMED_ACTION; + default -> throw new IllegalArgumentException("Unknown record type: " + recordType); }; return new NamedValue(actionName, fieldMap); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceFactory.java index 1dacb0fa37d9..ff74a2cceaab 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceFactory.java @@ -298,6 +298,24 @@ public String getSafeDescription() { } }; + private static final Resource CONNECTORS_RESOURCE = new Resource() { + + @Override + public String getIdentifier() { + return ResourceType.Connector.getValue(); + } + + @Override + public String getName() { + return "Connectors"; + } + + @Override + public String getSafeDescription() { + return "connectors"; + } + }; + /** * Gets the Resource for accessing the Controller. This includes Controller level configuration, bulletins, reporting tasks, and the cluster. * @@ -341,6 +359,14 @@ public static Resource getParameterProvidersResource() { return PARAMETER_PROVIDERS_RESOURCE; } + /** + * Gets the Resource for accessing Connectors. + * @return The resource for access Connectors + */ + public static Resource getConnectorsResource() { + return CONNECTORS_RESOURCE; + } + /** * Gets the Resource for accessing provenance. Access to this Resource allows the user to access data provenance. However, additional authorization * is required based on the component that generated the event and the attributes of the event. @@ -559,6 +585,7 @@ public String getSafeDescription() { case Label -> "Label"; case ParameterContext -> "Parameter Context"; case ParameterProvider -> "Parameter Provider"; + case Connector -> "Connector"; default -> "Component"; }; return componentType + " with ID " + identifier; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceType.java b/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceType.java index 0856573c9d13..33a29d880b49 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceType.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-authorization/src/main/java/org/apache/nifi/authorization/resource/ResourceType.java @@ -44,7 +44,8 @@ public enum ResourceType { Operation("/operation"), Tenant("/tenants"), ParameterContext("/parameter-contexts"), - ParameterProvider("/parameter-providers"); + ParameterProvider("/parameter-providers"), + Connector("/connectors"); final String value; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMapper.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMapper.java index f16d4eedc2d7..efa017ae5f77 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMapper.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMapper.java @@ -26,6 +26,12 @@ import org.apache.nifi.cluster.coordination.http.endpoints.ConnectionEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.ConnectionStatusEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.ConnectionsEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorFlowEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorPropertyGroupEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorPropertyGroupNamesEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorStatusEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorsEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.ControllerBulletinsEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.ControllerConfigurationEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.ControllerEndpointMerger; @@ -98,6 +104,7 @@ import org.apache.nifi.cluster.coordination.http.endpoints.UserGroupsEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.UsersEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.VerifyConfigEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.VerifyConnectorConfigStepEndpointMerger; import org.apache.nifi.cluster.manager.NodeResponse; import org.apache.nifi.stream.io.NullOutputStream; import org.apache.nifi.util.FormatUtils; @@ -138,6 +145,13 @@ public StandardHttpResponseMapper(final NiFiProperties nifiProperties) { endpointMergers.add(new ProcessorEndpointMerger()); endpointMergers.add(new ProcessorsEndpointMerger()); endpointMergers.add(new ProcessorRunStatusDetailsEndpointMerger()); + endpointMergers.add(new ConnectorEndpointMerger()); + endpointMergers.add(new ConnectorsEndpointMerger()); + endpointMergers.add(new ConnectorStatusEndpointMerger()); + endpointMergers.add(new ConnectorFlowEndpointMerger()); + endpointMergers.add(new ConnectorPropertyGroupEndpointMerger()); + endpointMergers.add(new ConnectorPropertyGroupNamesEndpointMerger()); + endpointMergers.add(new VerifyConnectorConfigStepEndpointMerger()); endpointMergers.add(new ConnectionEndpointMerger()); endpointMergers.add(new ConnectionsEndpointMerger()); endpointMergers.add(new PortEndpointMerger()); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/AssetsEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/AssetsEndpointMerger.java index 422ade1492d0..bcf5e0e06605 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/AssetsEndpointMerger.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/AssetsEndpointMerger.java @@ -31,7 +31,7 @@ public class AssetsEndpointMerger extends AbstractSingleDTOEndpoint> { - private static final Pattern ASSETS_URI = Pattern.compile("/nifi-api/parameter-contexts/[a-f0-9\\-]{36}/assets"); + private static final Pattern ASSETS_URI = Pattern.compile("/nifi-api/(parameter-contexts|connectors)/[a-f0-9\\-]{36}/assets"); @Override public boolean canHandle(final URI uri, final String method) { diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ComponentStateEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ComponentStateEndpointMerger.java index 60686c536973..02e06b93d85b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ComponentStateEndpointMerger.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ComponentStateEndpointMerger.java @@ -38,6 +38,8 @@ public class ComponentStateEndpointMerger extends AbstractSingleDTOEndpoint implements EndpointResponseMerger { + public static final Pattern CONNECTORS_URI_PATTERN = Pattern.compile("/nifi-api/connectors"); + public static final Pattern CONNECTOR_URI_PATTERN = Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}"); + public static final Pattern CONNECTOR_RUN_STATUS_URI_PATTERN = Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}/run-status"); + + @Override + public boolean canHandle(final URI uri, final String method) { + if (("GET".equalsIgnoreCase(method) || "PUT".equalsIgnoreCase(method)) && (CONNECTOR_URI_PATTERN.matcher(uri.getPath()).matches())) { + return true; + } else if ("PUT".equalsIgnoreCase(method) && CONNECTOR_RUN_STATUS_URI_PATTERN.matcher(uri.getPath()).matches()) { + return true; + } else if ("POST".equalsIgnoreCase(method) && CONNECTORS_URI_PATTERN.matcher(uri.getPath()).matches()) { + return true; + } + + return false; + } + + @Override + protected Class getEntityClass() { + return ConnectorEntity.class; + } + + @Override + protected void mergeResponses(final ConnectorEntity clientEntity, final Map entityMap, final Set successfulResponses, + final Set problematicResponses) { + + ConnectorEntityMerger.merge(clientEntity, entityMap); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorFlowEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorFlowEndpointMerger.java new file mode 100644 index 000000000000..a933ac469605 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorFlowEndpointMerger.java @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.cluster.coordination.http.endpoints; + +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.flow.ProcessGroupFlowDTO; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; + +import java.net.URI; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +public class ConnectorFlowEndpointMerger extends AbstractSingleDTOEndpoint { + public static final Pattern CONNECTOR_FLOW_URI_PATTERN = Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}/flow"); + + @Override + public boolean canHandle(final URI uri, final String method) { + return "GET".equalsIgnoreCase(method) && CONNECTOR_FLOW_URI_PATTERN.matcher(uri.getPath()).matches(); + } + + @Override + protected Class getEntityClass() { + return ProcessGroupFlowEntity.class; + } + + @Override + protected ProcessGroupFlowDTO getDto(final ProcessGroupFlowEntity entity) { + return entity.getProcessGroupFlow(); + } + + @Override + protected void mergeResponses(final ProcessGroupFlowDTO clientDto, final Map dtoMap, + final Set successfulResponses, final Set problematicResponses) { + + // Delegate to the shared flow merging logic + FlowMerger.mergeProcessGroupFlowDto(clientDto, dtoMap); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorPropertyGroupEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorPropertyGroupEndpointMerger.java new file mode 100644 index 000000000000..bfda6dcab871 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorPropertyGroupEndpointMerger.java @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.cluster.coordination.http.endpoints; + +import org.apache.nifi.cluster.coordination.http.EndpointResponseMerger; +import org.apache.nifi.cluster.manager.ConfigurationStepEntityMerger; +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; + +import java.net.URI; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +public class ConnectorPropertyGroupEndpointMerger extends AbstractSingleEntityEndpoint implements EndpointResponseMerger { + public static final Pattern CONNECTOR_PROPERTY_GROUP_URI_PATTERN = Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}/configuration-steps/[^/]+"); + + @Override + public boolean canHandle(final URI uri, final String method) { + return ("GET".equalsIgnoreCase(method) || "PUT".equalsIgnoreCase(method)) + && CONNECTOR_PROPERTY_GROUP_URI_PATTERN.matcher(uri.getPath()).matches(); + } + + @Override + protected Class getEntityClass() { + return ConfigurationStepEntity.class; + } + + @Override + protected void mergeResponses(final ConfigurationStepEntity clientEntity, final Map entityMap, + final Set successfulResponses, final Set problematicResponses) { + + ConfigurationStepEntityMerger.merge(clientEntity, entityMap); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorPropertyGroupNamesEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorPropertyGroupNamesEndpointMerger.java new file mode 100644 index 000000000000..fdc932c1bbf5 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorPropertyGroupNamesEndpointMerger.java @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.cluster.coordination.http.endpoints; + +import org.apache.nifi.cluster.coordination.http.EndpointResponseMerger; +import org.apache.nifi.cluster.manager.ConfigurationStepNamesEntityMerger; +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; + +import java.net.URI; +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +public class ConnectorPropertyGroupNamesEndpointMerger implements EndpointResponseMerger { + public static final Pattern CONNECTOR_PROPERTY_GROUPS_URI_PATTERN = Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}/configuration-steps"); + + @Override + public boolean canHandle(final URI uri, final String method) { + return "GET".equalsIgnoreCase(method) && CONNECTOR_PROPERTY_GROUPS_URI_PATTERN.matcher(uri.getPath()).matches(); + } + + @Override + public final NodeResponse merge(final URI uri, final String method, final Set successfulResponses, + final Set problematicResponses, final NodeResponse clientResponse) { + if (!canHandle(uri, method)) { + throw new IllegalArgumentException("Cannot use Endpoint Mapper of type " + getClass().getSimpleName() + + " to map responses for URI " + uri + ", HTTP Method " + method); + } + + final ConfigurationStepNamesEntity responseEntity = clientResponse.getClientResponse().readEntity(ConfigurationStepNamesEntity.class); + + final Map entityMap = new HashMap<>(); + for (final NodeResponse nodeResponse : successfulResponses) { + final ConfigurationStepNamesEntity nodeResponseEntity = nodeResponse == clientResponse + ? responseEntity + : nodeResponse.getClientResponse().readEntity(ConfigurationStepNamesEntity.class); + entityMap.put(nodeResponse.getNodeId(), nodeResponseEntity); + } + + ConfigurationStepNamesEntityMerger.merge(responseEntity, entityMap); + + // create a new client response + return new NodeResponse(clientResponse, responseEntity); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorStatusEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorStatusEndpointMerger.java new file mode 100644 index 000000000000..bff04b3fe719 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorStatusEndpointMerger.java @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.cluster.coordination.http.endpoints; + +import org.apache.nifi.cluster.manager.ComponentEntityStatusMerger; +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.manager.StatusMerger; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.status.NodeProcessGroupStatusSnapshotDTO; +import org.apache.nifi.web.api.dto.status.ProcessGroupStatusDTO; +import org.apache.nifi.web.api.entity.ProcessGroupStatusEntity; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +public class ConnectorStatusEndpointMerger extends AbstractSingleEntityEndpoint implements ComponentEntityStatusMerger { + public static final Pattern CONNECTOR_STATUS_URI_PATTERN = Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}/status"); + + @Override + public boolean canHandle(final URI uri, final String method) { + return "GET".equalsIgnoreCase(method) && CONNECTOR_STATUS_URI_PATTERN.matcher(uri.getPath()).matches(); + } + + @Override + protected Class getEntityClass() { + return ProcessGroupStatusEntity.class; + } + + @Override + protected void mergeResponses(final ProcessGroupStatusEntity clientEntity, final Map entityMap, + final Set successfulResponses, final Set problematicResponses) { + final ProcessGroupStatusDTO mergedProcessGroupStatus = clientEntity.getProcessGroupStatus(); + mergedProcessGroupStatus.setNodeSnapshots(new ArrayList<>()); + + final NodeIdentifier selectedNodeId = entityMap.entrySet().stream() + .filter(e -> e.getValue() == clientEntity) + .map(Map.Entry::getKey) + .findFirst() + .orElse(null); + + final NodeProcessGroupStatusSnapshotDTO selectedNodeSnapshot = new NodeProcessGroupStatusSnapshotDTO(); + selectedNodeSnapshot.setStatusSnapshot(mergedProcessGroupStatus.getAggregateSnapshot().clone()); + selectedNodeSnapshot.setAddress(selectedNodeId.getApiAddress()); + selectedNodeSnapshot.setApiPort(selectedNodeId.getApiPort()); + selectedNodeSnapshot.setNodeId(selectedNodeId.getId()); + + mergedProcessGroupStatus.getNodeSnapshots().add(selectedNodeSnapshot); + + for (final Map.Entry entry : entityMap.entrySet()) { + final NodeIdentifier nodeId = entry.getKey(); + final ProcessGroupStatusEntity nodeProcessGroupStatusEntity = entry.getValue(); + final ProcessGroupStatusDTO nodeProcessGroupStatus = nodeProcessGroupStatusEntity.getProcessGroupStatus(); + if (nodeProcessGroupStatus == mergedProcessGroupStatus) { + continue; + } + + mergeStatus(mergedProcessGroupStatus, clientEntity.getCanRead(), nodeProcessGroupStatus, nodeProcessGroupStatusEntity.getCanRead(), nodeId); + } + } + + @Override + public void mergeStatus(final ProcessGroupStatusDTO clientStatus, final boolean clientStatusReadablePermission, + final ProcessGroupStatusDTO status, final boolean statusReadablePermission, + final NodeIdentifier statusNodeIdentifier) { + StatusMerger.merge(clientStatus, clientStatusReadablePermission, status, statusReadablePermission, + statusNodeIdentifier.getId(), statusNodeIdentifier.getApiAddress(), statusNodeIdentifier.getApiPort()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorsEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorsEndpointMerger.java new file mode 100644 index 000000000000..9d89fb3b2d4d --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ConnectorsEndpointMerger.java @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.cluster.coordination.http.endpoints; + +import org.apache.nifi.cluster.coordination.http.EndpointResponseMerger; +import org.apache.nifi.cluster.manager.ConnectorsEntityMerger; +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorsEntity; + +import java.net.URI; +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +public class ConnectorsEndpointMerger implements EndpointResponseMerger { + public static final Pattern CONNECTORS_URI_PATTERN = Pattern.compile("/nifi-api/connectors"); + + @Override + public boolean canHandle(final URI uri, final String method) { + return "GET".equalsIgnoreCase(method) && CONNECTORS_URI_PATTERN.matcher(uri.getPath()).matches(); + } + + @Override + public final NodeResponse merge(final URI uri, final String method, final Set successfulResponses, final Set problematicResponses, final NodeResponse clientResponse) { + if (!canHandle(uri, method)) { + throw new IllegalArgumentException("Cannot use Endpoint Mapper of type " + getClass().getSimpleName() + " to map responses for URI " + uri + ", HTTP Method " + method); + } + + final ConnectorsEntity responseEntity = clientResponse.getClientResponse().readEntity(ConnectorsEntity.class); + final Set connectorEntities = responseEntity.getConnectors(); + + final Map> entityMap = new HashMap<>(); + for (final NodeResponse nodeResponse : successfulResponses) { + final ConnectorsEntity nodeResponseEntity = nodeResponse == clientResponse ? responseEntity : nodeResponse.getClientResponse().readEntity(ConnectorsEntity.class); + final Set nodeConnectorEntities = nodeResponseEntity.getConnectors(); + + for (final ConnectorEntity nodeConnectorEntity : nodeConnectorEntities) { + Map innerMap = entityMap.computeIfAbsent(nodeConnectorEntity.getId(), k -> new HashMap<>()); + innerMap.put(nodeResponse.getNodeId(), nodeConnectorEntity); + } + } + + ConnectorsEntityMerger.mergeConnectors(connectorEntities, entityMap); + + // create a new client response + return new NodeResponse(clientResponse, responseEntity); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java index f0344232a0c6..24754601bf39 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java @@ -55,6 +55,7 @@ protected void mergeResponses(final CurrentUserEntity clientEntity, final Map clientEntityComponentRestrictionsPermissions = clientEntity.getComponentRestrictionPermissions(); final Set entityComponentRestrictionsPermissions = entity.getComponentRestrictionPermissions(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/FlowMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/FlowMerger.java index ad15fa126954..cd84ddcdcec9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/FlowMerger.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/FlowMerger.java @@ -65,6 +65,17 @@ protected ProcessGroupFlowDTO getDto(final ProcessGroupFlowEntity entity) { protected void mergeResponses(final ProcessGroupFlowDTO clientDto, final Map dtoMap, final Set successfulResponses, final Set problematicResponses) { + mergeProcessGroupFlowDto(clientDto, dtoMap); + } + + /** + * Merges the ProcessGroupFlowDTO responses from all nodes into the client DTO. + * This method is package-private to allow reuse by other endpoint mergers (e.g., ConnectorFlowEndpointMerger). + * + * @param clientDto the client DTO to merge responses into + * @param dtoMap the responses from all nodes + */ + static void mergeProcessGroupFlowDto(final ProcessGroupFlowDTO clientDto, final Map dtoMap) { final FlowDTO flowDto = clientDto.getFlow(); final Set clientConnections = flowDto.getConnections(); final Set clientProcessors = flowDto.getProcessors(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/VerifyConnectorConfigStepEndpointMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/VerifyConnectorConfigStepEndpointMerger.java new file mode 100644 index 000000000000..9f32b505617e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/VerifyConnectorConfigStepEndpointMerger.java @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.coordination.http.endpoints; + +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.ConfigVerificationResultDTO; +import org.apache.nifi.web.api.dto.VerifyConnectorConfigStepRequestDTO; +import org.apache.nifi.web.api.entity.VerifyConnectorConfigStepRequestEntity; + +import java.net.URI; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +public class VerifyConnectorConfigStepEndpointMerger extends AbstractSingleEntityEndpoint { + public static final Pattern VERIFY_CONNECTOR_CONFIG_STEP_URI_PATTERN = + Pattern.compile("/nifi-api/connectors/[a-f0-9\\-]{36}/configuration-steps/[^/]+/verify-config(/[a-f0-9\\-]{36})?"); + + @Override + protected Class getEntityClass() { + return VerifyConnectorConfigStepRequestEntity.class; + } + + @Override + public boolean canHandle(final URI uri, final String method) { + return VERIFY_CONNECTOR_CONFIG_STEP_URI_PATTERN.matcher(uri.getPath()).matches(); + } + + @Override + protected void mergeResponses(final VerifyConnectorConfigStepRequestEntity clientEntity, + final Map entityMap, + final Set successfulResponses, + final Set problematicResponses) { + + final VerifyConnectorConfigStepRequestDTO requestDto = clientEntity.getRequest(); + final List results = requestDto.getResults(); + + if (results == null) { + return; + } + + final ConfigVerificationResultMerger resultMerger = new ConfigVerificationResultMerger(); + for (final Map.Entry entry : entityMap.entrySet()) { + final NodeIdentifier nodeId = entry.getKey(); + final VerifyConnectorConfigStepRequestEntity entity = entry.getValue(); + + final List nodeResults = entity.getRequest().getResults(); + resultMerger.addNodeResults(nodeId, nodeResults); + } + + final List aggregateResults = resultMerger.computeAggregateResults(); + + clientEntity.getRequest().setResults(aggregateResults); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ClusteredConnectorRequestReplicator.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ClusteredConnectorRequestReplicator.java new file mode 100644 index 000000000000..a3db59cf4f93 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ClusteredConnectorRequestReplicator.java @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.cluster.coordination.http.replication; + +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.Response.Status; +import jakarta.ws.rs.core.Response.Status.Family; +import jakarta.ws.rs.core.Response.StatusType; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.StandardNiFiUser; +import org.apache.nifi.cluster.coordination.ClusterCoordinator; +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.components.connector.ConnectorRequestReplicator; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.Entity; + +import java.io.IOException; +import java.net.URI; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.function.Supplier; + +public class ClusteredConnectorRequestReplicator implements ConnectorRequestReplicator { + private static final String GET = "GET"; + + private final Supplier requestReplicatorSupplier; + private final Supplier clusterCoordinatorSupplier; + private final String replicationScheme; + + public ClusteredConnectorRequestReplicator(final Supplier requestReplicatorSupplier, final Supplier clusterCoordinatorSupplier, + final boolean httpsEnabled) { + this.requestReplicatorSupplier = Objects.requireNonNull(requestReplicatorSupplier, "Request Replicator Supplier required"); + this.clusterCoordinatorSupplier = Objects.requireNonNull(clusterCoordinatorSupplier, "Cluster Coordinator Supplier required"); + this.replicationScheme = httpsEnabled ? "https" : "http"; + } + + @Override + public ConnectorState getState(final String connectorId) throws IOException { + final RequestReplicator requestReplicator = getRequestReplicator(); + final NiFiUser nodeUser = getNodeUser(); + final URI uri = URI.create(replicationScheme + "://localhost/nifi-api/connectors/" + connectorId); + final AsyncClusterResponse asyncResponse = requestReplicator.replicate(nodeUser, GET, uri, Map.of(), Map.of()); + + try { + final NodeResponse mergedNodeResponse = asyncResponse.awaitMergedResponse(); + final Response response = mergedNodeResponse.getClientResponse(); + verifyResponse(response.getStatusInfo(), connectorId); + + // Use the merged/updated entity if available, otherwise fall back to reading from the raw response. + // The updatedEntity contains the properly merged state from all nodes, while readEntity() would + // only return the state from whichever single node was selected as the "client response". + final ConnectorEntity connectorEntity; + final Entity updatedEntity = mergedNodeResponse.getUpdatedEntity(); + if (updatedEntity instanceof ConnectorEntity mergedConnectorEntity) { + connectorEntity = mergedConnectorEntity; + } else { + connectorEntity = response.readEntity(ConnectorEntity.class); + } + + final String stateName = connectorEntity.getComponent().getState(); + try { + return ConnectorState.valueOf(stateName); + } catch (final IllegalArgumentException e) { + throw new IOException("Received unknown Connector state '" + stateName + "' for Connector with ID " + connectorId); + } + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("Interrupted while waiting for response for status of Connector with ID " + connectorId, e); + } + } + + @Override + public void setFlowManager(final FlowManager flowManager) { + } + + private RequestReplicator getRequestReplicator() { + final RequestReplicator requestReplicator = requestReplicatorSupplier.get(); + return Objects.requireNonNull(requestReplicator, "Request Replicator required"); + } + + private NiFiUser getNodeUser() { + final ClusterCoordinator clusterCoordinator = clusterCoordinatorSupplier.get(); + Objects.requireNonNull(clusterCoordinator, "Cluster Coordinator required"); + + final NodeIdentifier localNodeIdentifier = clusterCoordinator.getLocalNodeIdentifier(); + Objects.requireNonNull(localNodeIdentifier, "Local Node Identifier required"); + + final Set nodeIdentities = localNodeIdentifier.getNodeIdentities(); + final String nodeIdentity = nodeIdentities.isEmpty() ? localNodeIdentifier.getApiAddress() : nodeIdentities.iterator().next(); + return new StandardNiFiUser.Builder().identity(nodeIdentity).build(); + } + + private void verifyResponse(final StatusType responseStatusType, final String connectorId) throws IOException { + final int statusCode = responseStatusType.getStatusCode(); + final String reason = responseStatusType.getReasonPhrase(); + + if (responseStatusType.getStatusCode() == Status.NOT_FOUND.getStatusCode()) { + throw new IllegalArgumentException("Connector with ID + " + connectorId + " does not exist"); + } + + final Family responseFamily = responseStatusType.getFamily(); + if (responseFamily == Family.SERVER_ERROR) { + throw new IOException("Server-side error requesting State for Connector with ID + " + connectorId + ". Status code: " + statusCode + ", reason: " + reason); + } + if (responseFamily == Family.CLIENT_ERROR) { + throw new IOException("Client-side error requesting State for Connector with ID " + connectorId + ". Status code: " + statusCode + ", reason: " + reason); + } + if (responseFamily != Family.SUCCESSFUL) { + throw new IOException("Unexpected response code " + statusCode + " returned when requesting State for Connector with ID " + connectorId + ": " + reason); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java index 5736b608246c..24bfda72225d 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java @@ -28,6 +28,7 @@ import org.apache.nifi.cluster.coordination.http.HttpResponseMapper; import org.apache.nifi.cluster.coordination.http.StandardHttpResponseMapper; import org.apache.nifi.cluster.coordination.http.endpoints.ConnectionEndpointMerger; +import org.apache.nifi.cluster.coordination.http.endpoints.ConnectorEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.ControllerServiceEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.FlowRegistryClientEndpointMerger; import org.apache.nifi.cluster.coordination.http.endpoints.FunnelEndpointMerger; @@ -670,6 +671,7 @@ private boolean isDeleteComponent(final String method, final String uriPath) { // Processors, which is done by issuing a request to DELETE /processors//threads return ConnectionEndpointMerger.CONNECTION_URI_PATTERN.matcher(uriPath).matches() || ProcessorEndpointMerger.PROCESSOR_URI_PATTERN.matcher(uriPath).matches() + || ConnectorEndpointMerger.CONNECTOR_URI_PATTERN.matcher(uriPath).matches() || FunnelEndpointMerger.FUNNEL_URI_PATTERN.matcher(uriPath).matches() || PortEndpointMerger.INPUT_PORT_URI_PATTERN.matcher(uriPath).matches() || PortEndpointMerger.OUTPUT_PORT_URI_PATTERN.matcher(uriPath).matches() diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConfigurationStepEntityMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConfigurationStepEntityMerger.java new file mode 100644 index 000000000000..4ac0f52e3cf3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConfigurationStepEntityMerger.java @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorPropertyDescriptorDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +public class ConfigurationStepEntityMerger { + + /** + * Merges the ConfigurationStepEntity responses. + * Note: Permissions are not merged here since they are managed at the parent Connector level. + * + * @param clientEntity the entity being returned to the client + * @param entityMap all node responses + */ + public static void merge(final ConfigurationStepEntity clientEntity, final Map entityMap) { + final ConfigurationStepConfigurationDTO clientDto = clientEntity.getConfigurationStep(); + final Map dtoMap = new HashMap<>(); + + for (final Map.Entry entry : entityMap.entrySet()) { + final ConfigurationStepEntity nodeConfigurationStepEntity = entry.getValue(); + final ConfigurationStepConfigurationDTO nodeConfigurationStepDto = nodeConfigurationStepEntity.getConfigurationStep(); + dtoMap.put(entry.getKey(), nodeConfigurationStepDto); + } + + mergeDtos(clientDto, dtoMap); + } + + private static void mergeDtos(final ConfigurationStepConfigurationDTO clientDto, final Map dtoMap) { + // if unauthorized for the client dto, simply return + if (clientDto == null) { + return; + } + + // Configuration step configurations should be consistent across all nodes + // We primarily need to ensure that the configuration step name is consistent + // The property values within property groups should also be consistent across nodes + + for (final Map.Entry nodeEntry : dtoMap.entrySet()) { + final ConfigurationStepConfigurationDTO nodeConfigurationStep = nodeEntry.getValue(); + + if (nodeConfigurationStep != null) { + // Verify that configuration step names are consistent + if (clientDto.getConfigurationStepName() != null && nodeConfigurationStep.getConfigurationStepName() != null) { + if (!clientDto.getConfigurationStepName().equals(nodeConfigurationStep.getConfigurationStepName())) { + // Log inconsistency but use the client's version + // In a properly functioning cluster, configuration step names should be consistent + } + } + + // Merge property group configurations + mergePropertyGroupConfigurations(clientDto.getPropertyGroupConfigurations(), nodeConfigurationStep.getPropertyGroupConfigurations()); + } + } + + // Merge property descriptors to handle dynamic allowable values + mergePropertyDescriptors(clientDto, dtoMap); + } + + private static void mergePropertyGroupConfigurations(final List clientPropertyGroups, final List nodePropertyGroups) { + if (clientPropertyGroups == null || nodePropertyGroups == null) { + return; + } + + // For property groups, we expect configurations to be consistent across nodes + // This merger primarily ensures that we have a consistent view + // Property values should not differ between nodes in a properly configured cluster + + for (final PropertyGroupConfigurationDTO clientPropertyGroup : clientPropertyGroups) { + for (final PropertyGroupConfigurationDTO nodePropertyGroup : nodePropertyGroups) { + if (clientPropertyGroup.getPropertyGroupName() != null + && clientPropertyGroup.getPropertyGroupName().equals(nodePropertyGroup.getPropertyGroupName())) { + + // Property values should be consistent across nodes + // In case of any inconsistency, we use the client's values + // This is primarily for defensive programming + break; + } + } + } + } + + /** + * Merges property descriptors across all nodes in the configuration step. + * This is necessary to handle dynamic allowable values that may differ between nodes. + * + * @param clientDto the configuration step DTO being returned to the client + * @param dtoMap all node configuration step DTOs + */ + static void mergePropertyDescriptors(final ConfigurationStepConfigurationDTO clientDto, final Map dtoMap) { + if (clientDto == null || clientDto.getPropertyGroupConfigurations() == null) { + return; + } + + for (final PropertyGroupConfigurationDTO clientPropertyGroup : clientDto.getPropertyGroupConfigurations()) { + if (clientPropertyGroup.getPropertyDescriptors() == null) { + continue; + } + + // Build a map of property descriptors from all nodes for this property group + final Map> propertyDescriptorMap = new HashMap<>(); + + for (final Map.Entry nodeEntry : dtoMap.entrySet()) { + final ConfigurationStepConfigurationDTO nodeConfigStep = nodeEntry.getValue(); + if (nodeConfigStep == null || nodeConfigStep.getPropertyGroupConfigurations() == null) { + continue; + } + + // Find the matching property group in the node's configuration + for (final PropertyGroupConfigurationDTO nodePropertyGroup : nodeConfigStep.getPropertyGroupConfigurations()) { + if (clientPropertyGroup.getPropertyGroupName() != null + && clientPropertyGroup.getPropertyGroupName().equals(nodePropertyGroup.getPropertyGroupName()) + && nodePropertyGroup.getPropertyDescriptors() != null) { + + // Aggregate property descriptors by name + for (final ConnectorPropertyDescriptorDTO propertyDescriptor : nodePropertyGroup.getPropertyDescriptors().values()) { + propertyDescriptorMap.computeIfAbsent(propertyDescriptor.getName(), name -> new HashMap<>()) + .put(nodeEntry.getKey(), propertyDescriptor); + } + } + } + } + + // Merge each property descriptor + for (final Map.Entry> entry : propertyDescriptorMap.entrySet()) { + final String propertyName = entry.getKey(); + final Map descriptorByNodeId = entry.getValue(); + + // Get the client's property descriptor directly from the map + final ConnectorPropertyDescriptorDTO clientPropertyDescriptor = clientPropertyGroup.getPropertyDescriptors().get(propertyName); + + if (clientPropertyDescriptor != null) { + ConnectorPropertyDescriptorDtoMerger.merge(clientPropertyDescriptor, descriptorByNodeId); + } + } + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConfigurationStepNamesEntityMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConfigurationStepNamesEntityMerger.java new file mode 100644 index 000000000000..66e541173df7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConfigurationStepNamesEntityMerger.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class ConfigurationStepNamesEntityMerger { + + /** + * Merges multiple ConfigurationStepNamesEntity responses. + * + * @param clientEntity the entity being returned to the client + * @param entityMap all node responses + */ + public static void merge(final ConfigurationStepNamesEntity clientEntity, final Map entityMap) { + if (clientEntity == null) { + return; + } + + final Set allConfigurationStepNames = new HashSet<>(); + + // Add client's configuration step names + if (clientEntity.getConfigurationStepNames() != null) { + allConfigurationStepNames.addAll(clientEntity.getConfigurationStepNames()); + } + + // Merge configuration step names from all nodes + for (final Map.Entry entry : entityMap.entrySet()) { + final ConfigurationStepNamesEntity nodeEntity = entry.getValue(); + + if (nodeEntity != null && nodeEntity.getConfigurationStepNames() != null) { + allConfigurationStepNames.addAll(nodeEntity.getConfigurationStepNames()); + } + } + + // Set the merged configuration step names (sorted for consistency) + final List sortedConfigurationStepNames = new ArrayList<>(allConfigurationStepNames); + sortedConfigurationStepNames.sort(String::compareTo); + clientEntity.setConfigurationStepNames(sortedConfigurationStepNames); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorEntityMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorEntityMerger.java new file mode 100644 index 000000000000..baa0f632367c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorEntityMerger.java @@ -0,0 +1,212 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; +import org.apache.nifi.web.api.dto.status.NodeConnectorStatusSnapshotDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Map; + +public class ConnectorEntityMerger { + /** + * Merges the ConnectorEntity responses. + * + * @param clientEntity the entity being returned to the client + * @param entityMap all node responses + */ + public static void merge(final ConnectorEntity clientEntity, final Map entityMap) { + final ConnectorDTO clientDto = clientEntity.getComponent(); + final Map dtoMap = new HashMap<>(); + for (final Map.Entry entry : entityMap.entrySet()) { + final ConnectorEntity nodeConnectorEntity = entry.getValue(); + final ConnectorDTO nodeConnectorDto = nodeConnectorEntity.getComponent(); + dtoMap.put(entry.getKey(), nodeConnectorDto); + } + + mergeDtos(clientDto, dtoMap); + + mergeStatus(clientEntity, entityMap); + } + + private static void mergeStatus(final ConnectorEntity clientEntity, final Map entityMap) { + final ConnectorStatusDTO mergedStatus = clientEntity.getStatus(); + if (mergedStatus == null) { + return; + } + + mergedStatus.setNodeSnapshots(new ArrayList<>()); + + // Identify the selected node (the one whose response was chosen as the client entity) + final NodeIdentifier selectedNodeId = entityMap.entrySet().stream() + .filter(e -> e.getValue() == clientEntity) + .map(Map.Entry::getKey) + .findFirst() + .orElse(null); + + // Add the selected node's snapshot to the node snapshots list + if (selectedNodeId != null && mergedStatus.getAggregateSnapshot() != null) { + final NodeConnectorStatusSnapshotDTO selectedNodeSnapshot = new NodeConnectorStatusSnapshotDTO(); + selectedNodeSnapshot.setStatusSnapshot(mergedStatus.getAggregateSnapshot().clone()); + selectedNodeSnapshot.setAddress(selectedNodeId.getApiAddress()); + selectedNodeSnapshot.setApiPort(selectedNodeId.getApiPort()); + selectedNodeSnapshot.setNodeId(selectedNodeId.getId()); + mergedStatus.getNodeSnapshots().add(selectedNodeSnapshot); + } + + // Merge snapshots from other nodes + for (final Map.Entry entry : entityMap.entrySet()) { + final NodeIdentifier nodeId = entry.getKey(); + final ConnectorEntity nodeEntity = entry.getValue(); + if (nodeEntity == clientEntity || nodeEntity == null) { + continue; + } + + final ConnectorStatusDTO nodeStatus = nodeEntity.getStatus(); + if (nodeStatus == null) { + continue; + } + + final boolean clientReadable = clientEntity.getPermissions() != null && Boolean.TRUE.equals(clientEntity.getPermissions().getCanRead()); + final boolean nodeReadable = nodeEntity.getPermissions() != null && Boolean.TRUE.equals(nodeEntity.getPermissions().getCanRead()); + StatusMerger.merge(mergedStatus, clientReadable, nodeStatus, nodeReadable, + nodeId.getId(), nodeId.getApiAddress(), nodeId.getApiPort()); + } + } + + private static void mergeDtos(final ConnectorDTO clientDto, final Map dtoMap) { + // if unauthorized for the client dto, simple return + if (clientDto == null) { + return; + } + + mergeState(clientDto, dtoMap); + + // Merge configuration steps to handle dynamic property descriptors + mergeActiveConfiguration(clientDto, dtoMap); + mergeWorkingConfiguration(clientDto, dtoMap); + } + + /** + * Merges state from cluster nodes using priority-based selection. + * Priority (highest to lowest): + * 1. UPDATE_FAILED - indicates a failed update that needs attention + * 2. PREPARING_FOR_UPDATE - connector is preparing for update + * 3. UPDATING - connector is actively updating + * 4. UPDATED - connector has been updated + * 5. STARTING/STOPPING - connector is transitioning between run states + * 6. RUNNING/STOPPED/DISABLED - stable states + */ + private static void mergeState(final ConnectorDTO clientDto, final Map dtoMap) { + String mergedState = clientDto.getState(); + + for (final ConnectorDTO nodeConnector : dtoMap.values()) { + if (nodeConnector != null) { + final String nodeState = nodeConnector.getState(); + if (nodeState != null) { + if (getStatePriority(nodeState) > getStatePriority(mergedState)) { + mergedState = nodeState; + } + } + } + } + + clientDto.setState(mergedState); + } + + private static int getStatePriority(final String state) { + if (state == null) { + return 0; + } + + final ConnectorState connectorState = ConnectorState.valueOf(state); + return switch (connectorState) { + case UPDATE_FAILED -> 7; + case PREPARING_FOR_UPDATE -> 6; + case UPDATING -> 5; + case DRAINING, PURGING -> 4; + case UPDATED -> 3; + case STARTING, STOPPING -> 2; + default -> 1; // RUNNING, STOPPED + }; + } + + private static void mergeActiveConfiguration(final ConnectorDTO clientDto, final Map dtoMap) { + final ConnectorConfigurationDTO clientConfig = clientDto.getActiveConfiguration(); + if (clientConfig == null || clientConfig.getConfigurationStepConfigurations() == null) { + return; + } + + // For each configuration step in the client's active configuration, merge it with the corresponding steps from all nodes + for (final ConfigurationStepConfigurationDTO clientStep : clientConfig.getConfigurationStepConfigurations()) { + final Map stepDtoMap = new HashMap<>(); + + for (final Map.Entry nodeEntry : dtoMap.entrySet()) { + final ConnectorDTO nodeDto = nodeEntry.getValue(); + if (nodeDto == null || nodeDto.getActiveConfiguration() == null || nodeDto.getActiveConfiguration().getConfigurationStepConfigurations() == null) { + continue; + } + + // Find the matching configuration step in the node's active configuration + nodeDto.getActiveConfiguration().getConfigurationStepConfigurations().stream() + .filter(step -> clientStep.getConfigurationStepName() != null + && clientStep.getConfigurationStepName().equals(step.getConfigurationStepName())) + .findFirst() + .ifPresent(step -> stepDtoMap.put(nodeEntry.getKey(), step)); + } + + // Merge property descriptors for this configuration step + ConfigurationStepEntityMerger.mergePropertyDescriptors(clientStep, stepDtoMap); + } + } + + private static void mergeWorkingConfiguration(final ConnectorDTO clientDto, final Map dtoMap) { + final ConnectorConfigurationDTO clientConfig = clientDto.getWorkingConfiguration(); + if (clientConfig == null || clientConfig.getConfigurationStepConfigurations() == null) { + return; + } + + // For each configuration step in the client's working configuration, merge it with the corresponding steps from all nodes + for (final ConfigurationStepConfigurationDTO clientStep : clientConfig.getConfigurationStepConfigurations()) { + final Map stepDtoMap = new HashMap<>(); + + for (final Map.Entry nodeEntry : dtoMap.entrySet()) { + final ConnectorDTO nodeDto = nodeEntry.getValue(); + if (nodeDto == null || nodeDto.getWorkingConfiguration() == null || nodeDto.getWorkingConfiguration().getConfigurationStepConfigurations() == null) { + continue; + } + + // Find the matching configuration step in the node's working configuration + nodeDto.getWorkingConfiguration().getConfigurationStepConfigurations().stream() + .filter(step -> clientStep.getConfigurationStepName() != null + && clientStep.getConfigurationStepName().equals(step.getConfigurationStepName())) + .findFirst() + .ifPresent(step -> stepDtoMap.put(nodeEntry.getKey(), step)); + } + + // Merge property descriptors for this configuration step + ConfigurationStepEntityMerger.mergePropertyDescriptors(clientStep, stepDtoMap); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorPropertyDescriptorDtoMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorPropertyDescriptorDtoMerger.java new file mode 100644 index 000000000000..ce6b29bffa1e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorPropertyDescriptorDtoMerger.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.AllowableValueDTO; +import org.apache.nifi.web.api.dto.ConnectorPropertyDescriptorDTO; +import org.apache.nifi.web.api.entity.AllowableValueEntity; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +public class ConnectorPropertyDescriptorDtoMerger { + public static void merge(final ConnectorPropertyDescriptorDTO clientPropertyDescriptor, final Map dtoMap) { + if (clientPropertyDescriptor.getAllowableValues() != null) { + final Map> allowableValueDtoToEntities = new LinkedHashMap<>(); + + addEntities(clientPropertyDescriptor, allowableValueDtoToEntities); + dtoMap.values().forEach(propertyDescriptorDTO -> addEntities(propertyDescriptorDTO, allowableValueDtoToEntities)); + + // Consider each allowable value from the client property descriptor. In the event of a duplicate allowable value, each entry will still + // be included in the merged list. Also ensure that each allowable value is present in all nodes property descriptors. + final List mergedAllowableValues = clientPropertyDescriptor.getAllowableValues().stream() + .map(allowableValueEntity -> allowableValueDtoToEntities.getOrDefault(allowableValueEntity.getAllowableValue(), Collections.emptyList())) + .filter(entities -> !entities.isEmpty() && allNodesHaveAllowableValue(entities.getFirst(), dtoMap)) + .map(AllowableValueEntityMerger::merge) + .collect(Collectors.toList()); + + clientPropertyDescriptor.setAllowableValues(mergedAllowableValues); + } + } + + private static boolean allNodesHaveAllowableValue(final AllowableValueEntity entity, final Map dtoMap) { + return dtoMap.values().stream() + .allMatch(propertyDescriptorDTO -> + propertyDescriptorDTO.getAllowableValues() != null && propertyDescriptorDTO.getAllowableValues().stream() + .anyMatch(allowableValueEntity -> allowableValueEntity.getAllowableValue().equals(entity.getAllowableValue()))); + } + + private static void addEntities(final ConnectorPropertyDescriptorDTO propertyDescriptorDTO, final Map> dtoToEntities) { + propertyDescriptorDTO.getAllowableValues().forEach( + allowableValueEntity -> dtoToEntities.computeIfAbsent(allowableValueEntity.getAllowableValue(), __ -> new ArrayList<>()).add(allowableValueEntity) + ); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorsEntityMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorsEntityMerger.java new file mode 100644 index 000000000000..a4d2e7692565 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/ConnectorsEntityMerger.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.entity.ConnectorEntity; + +import java.util.Map; +import java.util.Set; + +public class ConnectorsEntityMerger { + + private static final ConnectorEntityMerger connectorEntityMerger = new ConnectorEntityMerger(); + + /** + * Merges multiple ConnectorEntity responses. + * + * @param connectorEntities entities being returned to the client + * @param entityMap all node responses + */ + public static void mergeConnectors(final Set connectorEntities, final Map> entityMap) { + for (final ConnectorEntity entity : connectorEntities) { + connectorEntityMerger.merge(entity, entityMap.get(entity.getId())); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/StatusMerger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/StatusMerger.java index 06c2a970ba93..bd351db8fedb 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/StatusMerger.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/StatusMerger.java @@ -43,10 +43,13 @@ import org.apache.nifi.web.api.dto.status.ConnectionStatusDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusPredictionsSnapshotDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusSnapshotDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.ControllerServiceStatusDTO; import org.apache.nifi.web.api.dto.status.ControllerStatusDTO; import org.apache.nifi.web.api.dto.status.FlowAnalysisRuleStatusDTO; import org.apache.nifi.web.api.dto.status.NodeConnectionStatusSnapshotDTO; +import org.apache.nifi.web.api.dto.status.NodeConnectorStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.NodePortStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.NodeProcessGroupStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.NodeProcessorStatusSnapshotDTO; @@ -1075,4 +1078,96 @@ public static void merge(final FlowAnalysisRuleStatusDTO target, final FlowAnaly target.setValidationStatus(ValidationStatus.INVALID.name()); } } + + public static void merge(final ConnectorStatusDTO target, final boolean targetReadablePermission, final ConnectorStatusDTO toMerge, final boolean toMergeReadablePermission, + final String nodeId, final String nodeAddress, final Integer nodeApiPort) { + if (target == null || toMerge == null) { + return; + } + + if (targetReadablePermission && !toMergeReadablePermission) { + target.setGroupId(toMerge.getGroupId()); + target.setId(toMerge.getId()); + target.setName(toMerge.getName()); + target.setType(toMerge.getType()); + } + + if (ValidationStatus.VALIDATING.name().equalsIgnoreCase(toMerge.getValidationStatus())) { + target.setValidationStatus(ValidationStatus.VALIDATING.name()); + } else if (ValidationStatus.INVALID.name().equalsIgnoreCase(toMerge.getValidationStatus())) { + target.setValidationStatus(ValidationStatus.INVALID.name()); + } + + merge(target.getAggregateSnapshot(), targetReadablePermission, toMerge.getAggregateSnapshot(), toMergeReadablePermission); + + if (target.getNodeSnapshots() != null) { + final NodeConnectorStatusSnapshotDTO nodeSnapshot = new NodeConnectorStatusSnapshotDTO(); + nodeSnapshot.setStatusSnapshot(toMerge.getAggregateSnapshot()); + nodeSnapshot.setAddress(nodeAddress); + nodeSnapshot.setApiPort(nodeApiPort); + nodeSnapshot.setNodeId(nodeId); + + target.getNodeSnapshots().add(nodeSnapshot); + } + } + + public static void merge(final ConnectorStatusSnapshotDTO target, final boolean targetReadablePermission, + final ConnectorStatusSnapshotDTO toMerge, final boolean toMergeReadablePermission) { + if (target == null || toMerge == null) { + return; + } + + if (targetReadablePermission && !toMergeReadablePermission) { + target.setId(toMerge.getId()); + target.setGroupId(toMerge.getGroupId()); + target.setName(toMerge.getName()); + target.setType(toMerge.getType()); + } + + target.setFlowFilesSent(target.getFlowFilesSent() + toMerge.getFlowFilesSent()); + target.setBytesSent(target.getBytesSent() + toMerge.getBytesSent()); + target.setFlowFilesReceived(target.getFlowFilesReceived() + toMerge.getFlowFilesReceived()); + target.setBytesReceived(target.getBytesReceived() + toMerge.getBytesReceived()); + target.setBytesRead(target.getBytesRead() + toMerge.getBytesRead()); + target.setBytesWritten(target.getBytesWritten() + toMerge.getBytesWritten()); + + target.setFlowFilesQueued(target.getFlowFilesQueued() + toMerge.getFlowFilesQueued()); + target.setBytesQueued(target.getBytesQueued() + toMerge.getBytesQueued()); + + target.setActiveThreadCount(target.getActiveThreadCount() + toMerge.getActiveThreadCount()); + + // For idle status, the connector is considered idle only if ALL nodes report it as idle. + // The idle duration is the minimum across nodes (the most recently active node determines the duration). + if (Boolean.TRUE.equals(target.getIdle()) && Boolean.TRUE.equals(toMerge.getIdle())) { + if (target.getIdleDurationMillis() != null && toMerge.getIdleDurationMillis() != null) { + target.setIdleDurationMillis(Math.min(target.getIdleDurationMillis(), toMerge.getIdleDurationMillis())); + } else if (toMerge.getIdleDurationMillis() != null) { + target.setIdleDurationMillis(toMerge.getIdleDurationMillis()); + } + } else { + target.setIdle(false); + target.setIdleDurationMillis(null); + target.setIdleDuration(null); + } + + ProcessingPerformanceStatusMerger.mergeStatus(target.getProcessingPerformanceStatus(), toMerge.getProcessingPerformanceStatus()); + + updatePrettyPrintedFields(target); + } + + public static void updatePrettyPrintedFields(final ConnectorStatusSnapshotDTO target) { + target.setSent(prettyPrint(target.getFlowFilesSent(), target.getBytesSent())); + target.setReceived(prettyPrint(target.getFlowFilesReceived(), target.getBytesReceived())); + target.setRead(formatDataSize(target.getBytesRead())); + target.setWritten(formatDataSize(target.getBytesWritten())); + target.setQueued(prettyPrint(target.getFlowFilesQueued(), target.getBytesQueued())); + target.setQueuedCount(formatCount(target.getFlowFilesQueued())); + target.setQueuedSize(formatDataSize(target.getBytesQueued())); + + if (Boolean.TRUE.equals(target.getIdle()) && target.getIdleDurationMillis() != null) { + target.setIdleDuration(FormatUtils.formatHoursMinutesSeconds(target.getIdleDurationMillis(), TimeUnit.MILLISECONDS)); + } else { + target.setIdleDuration(null); + } + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/ConnectorRequestReplicatorConfiguration.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/ConnectorRequestReplicatorConfiguration.java new file mode 100644 index 000000000000..747b253716d6 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/ConnectorRequestReplicatorConfiguration.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.framework.configuration; + +import org.apache.nifi.cluster.coordination.ClusterCoordinator; +import org.apache.nifi.cluster.coordination.http.replication.ClusteredConnectorRequestReplicator; +import org.apache.nifi.cluster.coordination.http.replication.RequestReplicator; +import org.apache.nifi.components.connector.ConnectorRequestReplicator; +import org.apache.nifi.components.connector.StandaloneConnectorRequestReplicator; +import org.apache.nifi.util.NiFiProperties; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class ConnectorRequestReplicatorConfiguration { + + @Bean + public ConnectorRequestReplicator connectorRequestReplicator( + final NiFiProperties nifiProperties, + final ObjectProvider requestReplicatorProvider, + final ObjectProvider clusterCoordinatorProvider) { + + if (nifiProperties.isClustered()) { + // We have to use an ObjectProvider here and obtain a Supplier because of a circular dependency. + // The request replicator will not be available when created. However, it will be + // available before attempting to use the ConnectorRequestReplicator. + final String httpsHostname = nifiProperties.getProperty(NiFiProperties.WEB_HTTPS_HOST); + final boolean httpsEnabled = httpsHostname != null; + return new ClusteredConnectorRequestReplicator(requestReplicatorProvider::getIfAvailable, clusterCoordinatorProvider::getIfAvailable, httpsEnabled); + } + + return new StandaloneConnectorRequestReplicator(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/FrameworkClusterConfiguration.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/FrameworkClusterConfiguration.java index 1bdbde509377..b0eb1edd9ddb 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/FrameworkClusterConfiguration.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/framework/configuration/FrameworkClusterConfiguration.java @@ -48,8 +48,6 @@ public class FrameworkClusterConfiguration { private EventReporter eventReporter; - private FlowController flowController; - private ClusterCoordinator clusterCoordinator; private WebClientService webClientService; @@ -59,11 +57,6 @@ public void setProperties(final NiFiProperties properties) { this.properties = properties; } - @Autowired - public void setFlowController(final FlowController flowController) { - this.flowController = flowController; - } - @Autowired public void setEventReporter(final EventReporter eventReporter) { this.eventReporter = eventReporter; @@ -106,7 +99,7 @@ public ThreadPoolRequestReplicator requestReplicator( } @Bean - public ClusterDecommissionTask decommissionTask() { + public ClusterDecommissionTask decommissionTask(final FlowController flowController) { return new ClusterDecommissionTask(clusterCoordinator, flowController); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMergerTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMergerTest.java index 48d3ce0a51c1..2fdaff121bb8 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMergerTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMergerTest.java @@ -48,6 +48,7 @@ public void testMergeUserPermissions() { userNode1.setRestrictedComponentsPermissions(buildPermissions(false, false)); userNode1.setSystemPermissions(buildPermissions(true, true)); userNode1.setTenantsPermissions(buildPermissions(false, true)); + userNode1.setConnectorsPermissions(buildPermissions(true, false)); final Set componentRestrictionsNode1 = new HashSet<>(); componentRestrictionsNode1.add(buildComponentRestriction(RequiredPermission.ACCESS_KEYTAB, true, true)); @@ -64,6 +65,7 @@ public void testMergeUserPermissions() { userNode2.setRestrictedComponentsPermissions(buildPermissions(true, true)); userNode2.setSystemPermissions(buildPermissions(false, false)); userNode2.setTenantsPermissions(buildPermissions(true, true)); + userNode2.setConnectorsPermissions(buildPermissions(false, true)); final Set componentRestrictionsNode2 = new HashSet<>(); componentRestrictionsNode2.add(buildComponentRestriction(RequiredPermission.ACCESS_KEYTAB, true, false)); @@ -92,6 +94,8 @@ public void testMergeUserPermissions() { assertFalse(userNode1.getSystemPermissions().getCanWrite()); assertFalse(userNode1.getTenantsPermissions().getCanRead()); assertTrue(userNode1.getTenantsPermissions().getCanWrite()); + assertFalse(userNode1.getConnectorsPermissions().getCanRead()); + assertFalse(userNode1.getConnectorsPermissions().getCanWrite()); userNode1.getComponentRestrictionPermissions().forEach(componentRestriction -> { if (RequiredPermission.ACCESS_KEYTAB.getPermissionIdentifier().equals(componentRestriction.getRequiredPermission().getId())) { diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/replication/TestThreadPoolRequestReplicator.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/replication/TestThreadPoolRequestReplicator.java index 2baa903beaa6..808c65222932 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/replication/TestThreadPoolRequestReplicator.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/replication/TestThreadPoolRequestReplicator.java @@ -268,7 +268,7 @@ public void testMultipleRequestWithTwoPhaseCommit() throws Exception { when(coordinator.getConnectionStatus(Mockito.any(NodeIdentifier.class))).thenReturn(new NodeConnectionStatus(nodeId, NodeConnectionState.CONNECTED)); final AtomicInteger requestCount = new AtomicInteger(0); - final NiFiProperties props = NiFiProperties.createBasicNiFiProperties(null); + final NiFiProperties props = NiFiProperties.createBasicNiFiProperties((String) null); final MockReplicationClient client = new MockReplicationClient(); final RequestCompletionCallback requestCompletionCallback = (uri, method, responses) -> { @@ -329,7 +329,7 @@ public void testOneNodeRejectsTwoPhaseCommit() { final ClusterCoordinator coordinator = createClusterCoordinator(); final AtomicInteger requestCount = new AtomicInteger(0); - final NiFiProperties props = NiFiProperties.createBasicNiFiProperties(null); + final NiFiProperties props = NiFiProperties.createBasicNiFiProperties((String) null); final MockReplicationClient client = new MockReplicationClient(); final RequestCompletionCallback requestCompletionCallback = (uri, method, responses) -> { @@ -527,7 +527,7 @@ private void withReplicator(final WithReplicator function, final Status status, private void withReplicator(final WithReplicator function, final Status status, final long delayMillis, final RuntimeException failure, final String expectedRequestChain, final String expectedProxiedEntityGroups) { final ClusterCoordinator coordinator = createClusterCoordinator(); - final NiFiProperties nifiProps = NiFiProperties.createBasicNiFiProperties(null); + final NiFiProperties nifiProps = NiFiProperties.createBasicNiFiProperties((String) null); final MockReplicationClient client = new MockReplicationClient(); final RequestCompletionCallback requestCompletionCallback = (uri, method, responses) -> { }; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConfigurationStepEntityMergerTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConfigurationStepEntityMergerTest.java new file mode 100644 index 000000000000..d32e21bf7447 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConfigurationStepEntityMergerTest.java @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.AllowableValueDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorPropertyDescriptorDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.api.entity.AllowableValueEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; + +class ConfigurationStepEntityMergerTest { + + @Test + void testMergePropertyDescriptorsAcrossNodes() { + final ConfigurationStepEntity clientEntity = createConfigurationStepEntity("step1", + Map.of("hostname", List.of("localhost", "server1", "server2"))); + + final ConfigurationStepEntity node1Entity = createConfigurationStepEntity("step1", + Map.of("hostname", List.of("localhost", "server1"))); + + final ConfigurationStepEntity node2Entity = createConfigurationStepEntity("step1", + Map.of("hostname", List.of("localhost", "server1", "server2"))); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + entityMap.put(getNodeIdentifier("node2", 8002), node2Entity); + + ConfigurationStepEntityMerger.merge(clientEntity, entityMap); + + // Should only include allowable values present on all nodes + final PropertyGroupConfigurationDTO propertyGroup = clientEntity.getConfigurationStep().getPropertyGroupConfigurations().get(0); + final ConnectorPropertyDescriptorDTO descriptor = propertyGroup.getPropertyDescriptors().get("hostname"); + + assertNotNull(descriptor.getAllowableValues()); + assertEquals(2, descriptor.getAllowableValues().size()); + + final List values = descriptor.getAllowableValues().stream() + .map(AllowableValueEntity::getAllowableValue) + .map(AllowableValueDTO::getValue) + .sorted() + .toList(); + + assertEquals(List.of("localhost", "server1"), values); + } + + @Test + void testMergeWithNoOverlappingAllowableValues() { + final ConfigurationStepEntity clientEntity = createConfigurationStepEntity("step1", + Map.of("region", List.of("us-east-1", "us-west-1"))); + + final ConfigurationStepEntity node1Entity = createConfigurationStepEntity("step1", + Map.of("region", List.of("eu-west-1"))); + + final ConfigurationStepEntity node2Entity = createConfigurationStepEntity("step1", + Map.of("region", List.of("ap-south-1"))); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + entityMap.put(getNodeIdentifier("node2", 8002), node2Entity); + + ConfigurationStepEntityMerger.merge(clientEntity, entityMap); + + final PropertyGroupConfigurationDTO propertyGroup = clientEntity.getConfigurationStep().getPropertyGroupConfigurations().get(0); + final ConnectorPropertyDescriptorDTO descriptor = propertyGroup.getPropertyDescriptors().get("region"); + + assertNotNull(descriptor.getAllowableValues()); + assertEquals(0, descriptor.getAllowableValues().size()); + } + + @Test + void testMergeMultiplePropertyGroups() { + final ConfigurationStepConfigurationDTO clientStep = new ConfigurationStepConfigurationDTO(); + clientStep.setConfigurationStepName("database-config"); + + final PropertyGroupConfigurationDTO group1 = createPropertyGroup("connection", + Map.of("hostname", List.of("localhost", "server1"))); + final PropertyGroupConfigurationDTO group2 = createPropertyGroup("credentials", + Map.of("auth-type", List.of("password", "certificate"))); + + clientStep.setPropertyGroupConfigurations(List.of(group1, group2)); + + final ConfigurationStepConfigurationDTO node1Step = new ConfigurationStepConfigurationDTO(); + node1Step.setConfigurationStepName("database-config"); + final PropertyGroupConfigurationDTO node1Group1 = createPropertyGroup("connection", + Map.of("hostname", List.of("localhost", "server1"))); + final PropertyGroupConfigurationDTO node1Group2 = createPropertyGroup("credentials", + Map.of("auth-type", List.of("password"))); + node1Step.setPropertyGroupConfigurations(List.of(node1Group1, node1Group2)); + + final Map dtoMap = new HashMap<>(); + dtoMap.put(getNodeIdentifier("client", 8000), clientStep); + dtoMap.put(getNodeIdentifier("node1", 8001), node1Step); + + ConfigurationStepEntityMerger.mergePropertyDescriptors(clientStep, dtoMap); + + // Verify first property group + final ConnectorPropertyDescriptorDTO hostnameDescriptor = group1.getPropertyDescriptors().get("hostname"); + assertEquals(2, hostnameDescriptor.getAllowableValues().size()); + + // Verify second property group - should only have "password" since node1 doesn't have "certificate" + final ConnectorPropertyDescriptorDTO authDescriptor = group2.getPropertyDescriptors().get("auth-type"); + assertEquals(1, authDescriptor.getAllowableValues().size()); + assertEquals("password", authDescriptor.getAllowableValues().get(0).getAllowableValue().getValue()); + } + + private NodeIdentifier getNodeIdentifier(final String id, final int port) { + return new NodeIdentifier(id, "localhost", port, "localhost", port + 1, "localhost", port + 2, port + 3, true); + } + + private ConfigurationStepEntity createConfigurationStepEntity(final String stepName, final Map> propertyAllowableValues) { + final ConfigurationStepConfigurationDTO stepDto = new ConfigurationStepConfigurationDTO(); + stepDto.setConfigurationStepName(stepName); + stepDto.setConfigurationStepDescription("Test configuration step"); + + final PropertyGroupConfigurationDTO propertyGroup = createPropertyGroup("default", propertyAllowableValues); + stepDto.setPropertyGroupConfigurations(List.of(propertyGroup)); + + final ConfigurationStepEntity entity = new ConfigurationStepEntity(); + entity.setConfigurationStep(stepDto); + + return entity; + } + + private PropertyGroupConfigurationDTO createPropertyGroup(final String groupName, final Map> propertyAllowableValues) { + final PropertyGroupConfigurationDTO propertyGroup = new PropertyGroupConfigurationDTO(); + propertyGroup.setPropertyGroupName(groupName); + + final Map descriptors = new HashMap<>(); + for (final Map.Entry> entry : propertyAllowableValues.entrySet()) { + final ConnectorPropertyDescriptorDTO descriptor = new ConnectorPropertyDescriptorDTO(); + descriptor.setName(entry.getKey()); + descriptor.setDescription("Test property"); + descriptor.setType("STRING"); + descriptor.setRequired(false); + + final List allowableValues = entry.getValue().stream() + .map(value -> { + final AllowableValueDTO allowableValueDto = new AllowableValueDTO(); + allowableValueDto.setValue(value); + allowableValueDto.setDisplayName(value); + + final AllowableValueEntity entity = new AllowableValueEntity(); + entity.setAllowableValue(allowableValueDto); + entity.setCanRead(true); + return entity; + }) + .toList(); + + descriptor.setAllowableValues(allowableValues); + descriptors.put(entry.getKey(), descriptor); + } + + propertyGroup.setPropertyDescriptors(descriptors); + propertyGroup.setPropertyValues(new HashMap<>()); + + return propertyGroup; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConnectorEntityMergerTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConnectorEntityMergerTest.java new file mode 100644 index 000000000000..1560afa564bb --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConnectorEntityMergerTest.java @@ -0,0 +1,404 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.web.api.dto.AllowableValueDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.ConnectorPropertyDescriptorDTO; +import org.apache.nifi.web.api.dto.PermissionsDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.api.dto.RevisionDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusSnapshotDTO; +import org.apache.nifi.web.api.dto.status.NodeConnectorStatusSnapshotDTO; +import org.apache.nifi.web.api.entity.AllowableValueEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class ConnectorEntityMergerTest { + + @Test + void testMergeConnectorState() { + final ConnectorEntity clientEntity = createConnectorEntity("connector1", "STOPPED"); + final ConnectorEntity node1Entity = createConnectorEntity("connector1", "STOPPED"); + final ConnectorEntity node2Entity = createConnectorEntity("connector1", "STOPPED"); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + entityMap.put(getNodeIdentifier("node2", 8002), node2Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + assertEquals("STOPPED", clientEntity.getComponent().getState()); + } + + @Test + void testMergeConnectorStateWithNull() { + final ConnectorEntity clientEntity = createConnectorEntity("connector1", null); + final ConnectorEntity node1Entity = createConnectorEntity("connector1", "RUNNING"); + final ConnectorEntity node2Entity = createConnectorEntity("connector1", "RUNNING"); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + entityMap.put(getNodeIdentifier("node2", 8002), node2Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + assertEquals("RUNNING", clientEntity.getComponent().getState()); + } + + @Test + void testMergeConfigurationStepsWithDifferentAllowableValues() { + final ConnectorEntity clientEntity = createConnectorEntityWithConfig( + "connector1", "STOPPED", + Map.of("hostname", List.of("localhost", "server1", "server2")) + ); + + final ConnectorEntity node1Entity = createConnectorEntityWithConfig( + "connector1", "STOPPED", + Map.of("hostname", List.of("localhost", "server1")) + ); + + final ConnectorEntity node2Entity = createConnectorEntityWithConfig( + "connector1", "STOPPED", + Map.of("hostname", List.of("localhost", "server1", "server2")) + ); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + entityMap.put(getNodeIdentifier("node2", 8002), node2Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + // Verify working configuration was merged - should only have values present on all nodes + final ConnectorConfigurationDTO config = clientEntity.getComponent().getWorkingConfiguration(); + assertNotNull(config); + + final ConfigurationStepConfigurationDTO step = config.getConfigurationStepConfigurations().get(0); + final PropertyGroupConfigurationDTO propertyGroup = step.getPropertyGroupConfigurations().get(0); + final ConnectorPropertyDescriptorDTO descriptor = propertyGroup.getPropertyDescriptors().get("hostname"); + + assertNotNull(descriptor.getAllowableValues()); + assertEquals(2, descriptor.getAllowableValues().size()); + + final List values = descriptor.getAllowableValues().stream() + .map(AllowableValueEntity::getAllowableValue) + .map(AllowableValueDTO::getValue) + .sorted() + .toList(); + + assertEquals(List.of("localhost", "server1"), values); + } + + @Test + void testMergeMultipleConfigurationSteps() { + final ConnectorEntity clientEntity = createConnectorEntityWithMultipleSteps(); + final ConnectorEntity node1Entity = createConnectorEntityWithMultipleSteps(); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + final ConnectorConfigurationDTO config = clientEntity.getComponent().getWorkingConfiguration(); + assertNotNull(config); + assertEquals(2, config.getConfigurationStepConfigurations().size()); + } + + @Test + void testMergeConnectorStatusValidationStatusPriority() { + final ConnectorEntity clientEntity = createConnectorEntityWithStatus("connector1", "RUNNING", 1, "VALID", + 5, 100L, 10, 200L, 50L, 75L, 10, 1000L, true, 5000L); + final ConnectorEntity node1Entity = createConnectorEntityWithStatus("connector1", "RUNNING", 1, "VALIDATING", + 7, 150L, 12, 250L, 60L, 80L, 5, 500L, true, 3000L); + final ConnectorEntity node2Entity = createConnectorEntityWithStatus("connector1", "RUNNING", 1, "VALID", + 3, 200L, 8, 300L, 70L, 85L, 8, 800L, false, null); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + entityMap.put(getNodeIdentifier("node2", 8002), node2Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + assertEquals("VALIDATING", clientEntity.getStatus().getValidationStatus()); + } + + @Test + void testMergeStatusSnapshotsAggregated() { + final ConnectorEntity clientEntity = createConnectorEntityWithStatus("connector1", "RUNNING", 2, "VALID", + 5, 100L, 10, 200L, 50L, 75L, 10, 1000L, true, 5000L); + final ConnectorEntity node1Entity = createConnectorEntityWithStatus("connector1", "RUNNING", 3, "VALID", + 7, 150L, 12, 250L, 60L, 80L, 5, 500L, true, 3000L); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + final ConnectorStatusDTO mergedStatus = clientEntity.getStatus(); + assertNotNull(mergedStatus); + assertNotNull(mergedStatus.getAggregateSnapshot()); + assertNotNull(mergedStatus.getNodeSnapshots()); + + // Verify node snapshots contain both nodes + assertEquals(2, mergedStatus.getNodeSnapshots().size()); + + // Verify aggregate snapshot has summed values + final ConnectorStatusSnapshotDTO aggregate = mergedStatus.getAggregateSnapshot(); + assertEquals(Integer.valueOf(12), aggregate.getFlowFilesSent()); // 5 + 7 + assertEquals(Long.valueOf(250L), aggregate.getBytesSent()); // 100 + 150 + assertEquals(Integer.valueOf(22), aggregate.getFlowFilesReceived()); // 10 + 12 + assertEquals(Long.valueOf(450L), aggregate.getBytesReceived()); // 200 + 250 + assertEquals(Long.valueOf(110L), aggregate.getBytesRead()); // 50 + 60 + assertEquals(Long.valueOf(155L), aggregate.getBytesWritten()); // 75 + 80 + assertEquals(Integer.valueOf(15), aggregate.getFlowFilesQueued()); // 10 + 5 + assertEquals(Long.valueOf(1500L), aggregate.getBytesQueued()); // 1000 + 500 + assertEquals(Integer.valueOf(5), aggregate.getActiveThreadCount()); // 2 + 3 + + // Both nodes are idle, so aggregate should be idle with min duration + assertTrue(aggregate.getIdle()); + assertEquals(Long.valueOf(3000L), aggregate.getIdleDurationMillis()); // min(5000, 3000) + } + + @Test + void testMergeStatusSnapshotsIdleWhenOneNodeNotIdle() { + final ConnectorEntity clientEntity = createConnectorEntityWithStatus("connector1", "RUNNING", 2, "VALID", + 5, 100L, 10, 200L, 50L, 75L, 10, 1000L, true, 5000L); + final ConnectorEntity node1Entity = createConnectorEntityWithStatus("connector1", "RUNNING", 3, "VALID", + 7, 150L, 12, 250L, 60L, 80L, 5, 500L, false, null); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + final ConnectorStatusSnapshotDTO aggregate = clientEntity.getStatus().getAggregateSnapshot(); + + // One node is not idle, so aggregate should not be idle + assertFalse(aggregate.getIdle()); + assertNull(aggregate.getIdleDurationMillis()); + } + + @Test + void testMergeStatusNodeSnapshotsContainCorrectNodeInfo() { + final ConnectorEntity clientEntity = createConnectorEntityWithStatus("connector1", "RUNNING", 2, "VALID", + 5, 100L, 10, 200L, 50L, 75L, 10, 1000L, false, null); + final ConnectorEntity node1Entity = createConnectorEntityWithStatus("connector1", "RUNNING", 3, "VALID", + 7, 150L, 12, 250L, 60L, 80L, 5, 500L, false, null); + + final NodeIdentifier clientNodeId = getNodeIdentifier("client", 8000); + final NodeIdentifier node1NodeId = getNodeIdentifier("node1", 8001); + + final Map entityMap = new HashMap<>(); + entityMap.put(clientNodeId, clientEntity); + entityMap.put(node1NodeId, node1Entity); + + ConnectorEntityMerger.merge(clientEntity, entityMap); + + final List nodeSnapshots = clientEntity.getStatus().getNodeSnapshots(); + assertEquals(2, nodeSnapshots.size()); + + // Find the client node snapshot + final NodeConnectorStatusSnapshotDTO clientSnapshot = nodeSnapshots.stream() + .filter(s -> s.getNodeId().equals(clientNodeId.getId())) + .findFirst() + .orElse(null); + assertNotNull(clientSnapshot); + assertEquals(clientNodeId.getApiAddress(), clientSnapshot.getAddress()); + assertEquals(clientNodeId.getApiPort(), clientSnapshot.getApiPort()); + assertEquals(Long.valueOf(100L), clientSnapshot.getStatusSnapshot().getBytesSent()); + + // Find the other node snapshot + final NodeConnectorStatusSnapshotDTO node1Snapshot = nodeSnapshots.stream() + .filter(s -> s.getNodeId().equals(node1NodeId.getId())) + .findFirst() + .orElse(null); + assertNotNull(node1Snapshot); + assertEquals(node1NodeId.getApiAddress(), node1Snapshot.getAddress()); + assertEquals(node1NodeId.getApiPort(), node1Snapshot.getApiPort()); + assertEquals(Long.valueOf(150L), node1Snapshot.getStatusSnapshot().getBytesSent()); + } + + @Test + void testMergeStatusWithNullStatus() { + final ConnectorEntity clientEntity = createConnectorEntity("connector1", "STOPPED"); + final ConnectorEntity node1Entity = createConnectorEntity("connector1", "STOPPED"); + + final Map entityMap = new HashMap<>(); + entityMap.put(getNodeIdentifier("client", 8000), clientEntity); + entityMap.put(getNodeIdentifier("node1", 8001), node1Entity); + + // Both have null status - should not throw + ConnectorEntityMerger.merge(clientEntity, entityMap); + assertNull(clientEntity.getStatus()); + } + + private NodeIdentifier getNodeIdentifier(final String id, final int port) { + return new NodeIdentifier(id, "localhost", port, "localhost", port + 1, "localhost", port + 2, port + 3, true); + } + + private ConnectorEntity createConnectorEntity(final String id, final String state) { + final ConnectorDTO dto = new ConnectorDTO(); + dto.setId(id); + dto.setName("Test Connector"); + dto.setType("TestConnector"); + dto.setState(state); + + final PermissionsDTO permissions = new PermissionsDTO(); + permissions.setCanRead(true); + permissions.setCanWrite(true); + + final ConnectorEntity entity = new ConnectorEntity(); + entity.setComponent(dto); + entity.setRevision(createNewRevision()); + entity.setPermissions(permissions); + + return entity; + } + + private ConnectorEntity createConnectorEntityWithStatus(final String id, final String state, final int activeThreadCount, final String validationStatus, + final Integer flowFilesSent, final Long bytesSent, + final Integer flowFilesReceived, final Long bytesReceived, + final Long bytesRead, final Long bytesWritten, + final Integer flowFilesQueued, final Long bytesQueued, + final Boolean idle, final Long idleDurationMillis) { + final ConnectorEntity entity = createConnectorEntity(id, state); + + final ConnectorStatusSnapshotDTO snapshot = new ConnectorStatusSnapshotDTO(); + snapshot.setId(id); + snapshot.setName("Test Connector"); + snapshot.setType("TestConnector"); + snapshot.setRunStatus(state); + snapshot.setActiveThreadCount(activeThreadCount); + snapshot.setFlowFilesSent(flowFilesSent); + snapshot.setBytesSent(bytesSent); + snapshot.setFlowFilesReceived(flowFilesReceived); + snapshot.setBytesReceived(bytesReceived); + snapshot.setBytesRead(bytesRead); + snapshot.setBytesWritten(bytesWritten); + snapshot.setFlowFilesQueued(flowFilesQueued); + snapshot.setBytesQueued(bytesQueued); + snapshot.setIdle(idle); + snapshot.setIdleDurationMillis(idleDurationMillis); + + final ConnectorStatusDTO status = new ConnectorStatusDTO(); + status.setId(id); + status.setRunStatus(state); + status.setValidationStatus(validationStatus); + status.setAggregateSnapshot(snapshot); + entity.setStatus(status); + + return entity; + } + + private ConnectorEntity createConnectorEntityWithConfig(final String id, final String state, + final Map> propertyAllowableValues) { + final ConnectorEntity entity = createConnectorEntity(id, state); + + final ConnectorConfigurationDTO config = new ConnectorConfigurationDTO(); + final ConfigurationStepConfigurationDTO step = createConfigurationStep("step1", propertyAllowableValues); + config.setConfigurationStepConfigurations(List.of(step)); + + entity.getComponent().setActiveConfiguration(config); + entity.getComponent().setWorkingConfiguration(config); + + return entity; + } + + private ConnectorEntity createConnectorEntityWithMultipleSteps() { + final ConnectorEntity entity = createConnectorEntity("connector1", "STOPPED"); + + final ConnectorConfigurationDTO config = new ConnectorConfigurationDTO(); + final ConfigurationStepConfigurationDTO step1 = createConfigurationStep("step1", + Map.of("hostname", List.of("localhost"))); + final ConfigurationStepConfigurationDTO step2 = createConfigurationStep("step2", + Map.of("port", List.of("8080", "9090"))); + config.setConfigurationStepConfigurations(List.of(step1, step2)); + + entity.getComponent().setActiveConfiguration(config); + entity.getComponent().setWorkingConfiguration(config); + + return entity; + } + + private ConfigurationStepConfigurationDTO createConfigurationStep(final String stepName, + final Map> propertyAllowableValues) { + final ConfigurationStepConfigurationDTO step = new ConfigurationStepConfigurationDTO(); + step.setConfigurationStepName(stepName); + step.setConfigurationStepDescription("Test step"); + + final PropertyGroupConfigurationDTO propertyGroup = new PropertyGroupConfigurationDTO(); + propertyGroup.setPropertyGroupName("default"); + + final Map descriptors = new HashMap<>(); + for (final Map.Entry> entry : propertyAllowableValues.entrySet()) { + final ConnectorPropertyDescriptorDTO descriptor = new ConnectorPropertyDescriptorDTO(); + descriptor.setName(entry.getKey()); + descriptor.setDescription("Test property"); + descriptor.setType("STRING"); + + final List allowableValues = entry.getValue().stream() + .map(value -> { + final AllowableValueDTO allowableValueDto = new AllowableValueDTO(); + allowableValueDto.setValue(value); + + final AllowableValueEntity allowableEntity = new AllowableValueEntity(); + allowableEntity.setAllowableValue(allowableValueDto); + allowableEntity.setCanRead(true); + return allowableEntity; + }) + .toList(); + + descriptor.setAllowableValues(allowableValues); + descriptors.put(entry.getKey(), descriptor); + } + + propertyGroup.setPropertyDescriptors(descriptors); + propertyGroup.setPropertyValues(new HashMap<>()); + + step.setPropertyGroupConfigurations(List.of(propertyGroup)); + + return step; + } + + private RevisionDTO createNewRevision() { + final RevisionDTO revisionDto = new RevisionDTO(); + revisionDto.setClientId(getClass().getName()); + revisionDto.setVersion(0L); + return revisionDto; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConnectorPropertyDescriptorDtoMergerTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConnectorPropertyDescriptorDtoMergerTest.java new file mode 100644 index 000000000000..b2c4ad25277e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/manager/ConnectorPropertyDescriptorDtoMergerTest.java @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.cluster.manager; + +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.util.EqualsWrapper; +import org.apache.nifi.web.api.dto.AllowableValueDTO; +import org.apache.nifi.web.api.dto.ConnectorPropertyDescriptorDTO; +import org.apache.nifi.web.api.entity.AllowableValueEntity; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.function.Function; +import java.util.stream.Collectors; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; + +class ConnectorPropertyDescriptorDtoMergerTest { + + @Test + void testMergeWithNoAllowableValues() { + final ConnectorPropertyDescriptorDTO clientPropertyDescriptor = new ConnectorPropertyDescriptorDTO(); + + final Map dtoMap = + Map.of(createNodeIdentifier("node1"), new ConnectorPropertyDescriptorDTO(), + createNodeIdentifier("node2"), new ConnectorPropertyDescriptorDTO()); + + ConnectorPropertyDescriptorDtoMerger.merge(clientPropertyDescriptor, dtoMap); + + assertNull(clientPropertyDescriptor.getAllowableValues()); + } + + @Test + void testMergeWithEmptyAllowableValuesList() { + testMerge( + createPropertyDescriptorDTO(), + Map.of(createNodeIdentifier("node1"), createPropertyDescriptorDTO(), + createNodeIdentifier("node2"), createPropertyDescriptorDTO()), + createPropertyDescriptorDTO() + ); + } + + @Test + void testMergeWithSingleNode() { + testMerge( + createPropertyDescriptorDTO(v("value1"), v("value2")), + Collections.emptyMap(), + createPropertyDescriptorDTO(v("value1"), v("value2")) + ); + } + + @Test + void testMergeWithNonOverlappingAllowableValues() { + testMerge( + createPropertyDescriptorDTO(v("value1"), v("value2")), + Map.of(createNodeIdentifier("node1"), createPropertyDescriptorDTO(v("value3")), + createNodeIdentifier("node2"), createPropertyDescriptorDTO(v("value4"), v("value5"), v("value6"))), + createPropertyDescriptorDTO() + ); + } + + @Test + void testMergeWithOverlappingAllowableValues() { + testMerge( + createPropertyDescriptorDTO(v("value1"), v("value2"), v("value3")), + Map.of(createNodeIdentifier("node1"), createPropertyDescriptorDTO(v("value1"), v("value2"), v("value3")), + createNodeIdentifier("node2"), createPropertyDescriptorDTO(v("value2"), v("value3", false))), + createPropertyDescriptorDTO(v("value2"), v("value3", false)) + ); + } + + @Test + void testMergeWithIdenticalAllowableValues() { + testMerge( + createPropertyDescriptorDTO(v("value1"), v("value2")), + Map.of(createNodeIdentifier("node1"), createPropertyDescriptorDTO(v("value1"), v("value2")), + createNodeIdentifier("node2"), createPropertyDescriptorDTO(v("value1"), v("value2"))), + createPropertyDescriptorDTO(v("value1"), v("value2")) + ); + } + + @Test + void testMergeWithDuplicateAllowableValues() { + testMerge( + createPropertyDescriptorDTO(v("value1"), v("value1")), + Map.of(createNodeIdentifier("node1"), createPropertyDescriptorDTO(v("value1"), v("value1")), + createNodeIdentifier("node2"), createPropertyDescriptorDTO(v("value1"), v("value1"))), + createPropertyDescriptorDTO(v("value1"), v("value1")) + ); + } + + private ConnectorPropertyDescriptorDTO createPropertyDescriptorDTO(final AllowableValueData... allowableValueData) { + final ConnectorPropertyDescriptorDTO clientPropertyDescriptor = new ConnectorPropertyDescriptorDTO(); + + final List allowableValueEntities = Arrays.stream(allowableValueData) + .map(AllowableValueData::toEntity) + .collect(Collectors.toList()); + + clientPropertyDescriptor.setAllowableValues(allowableValueEntities); + + return clientPropertyDescriptor; + } + + private NodeIdentifier createNodeIdentifier(final String id) { + return new NodeIdentifier(id, id, 1, id, 1, id, 1, null, false); + } + + private void testMerge(final ConnectorPropertyDescriptorDTO clientPropertyDescriptor, + final Map dtoMap, + final ConnectorPropertyDescriptorDTO expected) { + ConnectorPropertyDescriptorDtoMerger.merge(clientPropertyDescriptor, dtoMap); + + final List> equalsProperties = Arrays.asList( + AllowableValueEntity::getAllowableValue, + AllowableValueEntity::getCanRead + ); + + final List> expectedWrappers = EqualsWrapper.wrapList(expected.getAllowableValues(), equalsProperties); + final List> actualWrappers = EqualsWrapper.wrapList(clientPropertyDescriptor.getAllowableValues(), equalsProperties); + + assertEquals(expectedWrappers, actualWrappers); + } + + private static class AllowableValueData { + private final String value; + private final Boolean canRead; + + private AllowableValueData(final String value, final Boolean canRead) { + this.value = value; + this.canRead = canRead; + } + + private AllowableValueEntity toEntity() { + final AllowableValueEntity entity = new AllowableValueEntity(); + + final AllowableValueDTO allowableValueDTO = new AllowableValueDTO(); + allowableValueDTO.setValue(value); + + entity.setAllowableValue(allowableValueDTO); + entity.setCanRead(canRead); + + return entity; + } + } + + private static AllowableValueData v(final String value) { + return v(value, true); + } + + private static AllowableValueData v(final String value, final Boolean canRead) { + return new AllowableValueData(value, canRead); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAsset.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAsset.java index 77e2ef5b64de..ed0cea997b63 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAsset.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAsset.java @@ -23,14 +23,14 @@ public class StandardAsset implements Asset { private final String identifier; - private final String parameterContextIdentifier; + private final String ownerIdentifier; private final String name; private final File file; private final String digest; - public StandardAsset(final String identifier, final String paramContextIdentifier, final String name, final File file, final String digest) { + public StandardAsset(final String identifier, final String ownerIdentifier, final String name, final File file, final String digest) { this.identifier = Objects.requireNonNull(identifier, "Identifier is required"); - this.parameterContextIdentifier = Objects.requireNonNull(paramContextIdentifier, "Parameter Context Identifier is required"); + this.ownerIdentifier = Objects.requireNonNull(ownerIdentifier, "Owner Identifier is required"); this.name = Objects.requireNonNull(name, "Name is required"); this.file = Objects.requireNonNull(file, "File is required"); this.digest = digest; @@ -43,7 +43,12 @@ public String getIdentifier() { @Override public String getParameterContextIdentifier() { - return parameterContextIdentifier; + return ownerIdentifier; + } + + @Override + public String getOwnerIdentifier() { + return ownerIdentifier; } @Override diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAssetManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAssetManager.java index f643cccf3a73..28b905462216 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAssetManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAssetManager.java @@ -68,36 +68,20 @@ public void initialize(final AssetManagerInitializationContext context) { } @Override - public Asset createAsset(final String parameterContextId, final String assetName, final InputStream contents) throws IOException { - final String assetId = createAssetId(parameterContextId, assetName); - - final File file = getFile(parameterContextId, assetName); - final File dir = file.getParentFile(); - if (!dir.exists()) { - try { - Files.createDirectories(dir.toPath()); - } catch (final IOException ioe) { - throw new IOException("Could not create directory in order to store asset", ioe); - } - } - - // Write contents to a temporary file, then move it to the final location. - // This allows us to avoid a situation where we upload a file, then we attempt to overwrite it but fail, leaving a corrupt asset. - final File tempFile = new File(dir, file.getName() + ".tmp"); - logger.debug("Writing temp asset file [{}]", tempFile.getAbsolutePath()); + public Asset createAsset(final String ownerId, final String assetName, final InputStream contents) throws IOException { + final String assetId = createAssetId(ownerId, assetName); + final File assetFile = getFile(ownerId, assetName); + return saveAsset(ownerId, assetId, assetName, contents, assetFile); + } - try { - Files.copy(contents, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING); - } catch (final Exception e) { - throw new IOException("Failed to write asset to file " + tempFile.getAbsolutePath(), e); + @Override + public Asset saveAsset(final String ownerId, final String assetId, final String assetName, final InputStream contents) throws IOException { + final String generatedAssetId = createAssetId(ownerId, assetName); + if (!generatedAssetId.equals(assetId)) { + throw new IllegalArgumentException("The provided asset id [%s] does not match the id this asset manager generated for the given owner and name".formatted(assetId)); } - - Files.move(tempFile.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING); - - final String digest = computeDigest(file); - final Asset asset = new StandardAsset(assetId, parameterContextId, assetName, file, digest); - assets.put(assetId, asset); - return asset; + final File assetFile = getFile(ownerId, assetName); + return saveAsset(ownerId, assetId, assetName, contents, assetFile); } @Override @@ -106,22 +90,22 @@ public Optional getAsset(final String id) { } @Override - public List getAssets(final String parameterContextId) { + public List getAssets(final String ownerId) { final List allAssets = new ArrayList<>(assets.values()); - final List paramContextAssets = new ArrayList<>(); + final List ownerAssets = new ArrayList<>(); for (final Asset asset : allAssets) { - if (asset.getParameterContextIdentifier().equals(parameterContextId)) { - paramContextAssets.add(asset); + if (asset.getOwnerIdentifier().equals(ownerId)) { + ownerAssets.add(asset); } } - return paramContextAssets; + return ownerAssets; } @Override - public Asset createMissingAsset(final String parameterContextId, final String assetName) { - final String assetId = createAssetId(parameterContextId, assetName); - final File file = getFile(parameterContextId, assetName); - final Asset asset = new StandardAsset(assetId, parameterContextId, assetName, file, null); + public Asset createMissingAsset(final String ownerId, final String assetName) { + final String assetId = createAssetId(ownerId, assetName); + final File file = getFile(ownerId, assetName); + final Asset asset = new StandardAsset(assetId, ownerId, assetName, file, null); assets.put(assetId, asset); return asset; } @@ -155,14 +139,43 @@ public Optional deleteAsset(final String id) { return Optional.of(removed); } - private String createAssetId(final String parameterContextId, final String assetName) { - final String seed = parameterContextId + "/" + assetName; + private Asset saveAsset(final String ownerId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException { + final File dir = assetFile.getParentFile(); + if (!dir.exists()) { + try { + Files.createDirectories(dir.toPath()); + } catch (final IOException ioe) { + throw new IOException("Could not create directory in order to store asset", ioe); + } + } + + // Write contents to a temporary file, then move it to the final location. + // This allows us to avoid a situation where we upload a file, then we attempt to overwrite it but fail, leaving a corrupt asset. + final File tempFile = new File(dir, assetFile.getName() + ".tmp"); + logger.debug("Writing temp asset file [{}]", tempFile.getAbsolutePath()); + + try { + Files.copy(contents, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING); + } catch (final Exception e) { + throw new IOException("Failed to write asset to file " + tempFile.getAbsolutePath(), e); + } + + Files.move(tempFile.toPath(), assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING); + + final String digest = computeDigest(assetFile); + final Asset asset = new StandardAsset(assetId, ownerId, assetName, assetFile, digest); + assets.put(assetId, asset); + return asset; + } + + private String createAssetId(final String ownerId, final String assetName) { + final String seed = ownerId + "/" + assetName; return UUID.nameUUIDFromBytes(seed.getBytes(StandardCharsets.UTF_8)).toString(); } - private File getFile(final String paramContextId, final String assetName) { + private File getFile(final String ownerId, final String assetName) { final Path parentPath = assetStorageLocation.toPath().normalize(); - final Path assetPath = Paths.get(paramContextId, assetName).normalize(); + final Path assetPath = Paths.get(ownerId, assetName).normalize(); final Path fullPath = parentPath.resolve(assetPath); return fullPath.toFile(); } @@ -183,17 +196,17 @@ private void recoverLocalAssets() throws IOException { continue; } - final String contextId = file.getName(); + final String ownerId = file.getName(); final File[] assetFiles = file.listFiles(); if (assetFiles == null) { - logger.warn("Unable to determine which assets exist for Parameter Context {}", contextId); + logger.warn("Unable to determine which assets exist for owner {}", ownerId); continue; } for (final File assetFile : assetFiles) { - final String assetId = createAssetId(contextId, assetFile.getName()); + final String assetId = createAssetId(ownerId, assetFile.getName()); final String digest = computeDigest(assetFile); - final Asset asset = new StandardAsset(assetId, contextId, assetFile.getName(), assetFile, digest); + final Asset asset = new StandardAsset(assetId, ownerId, assetFile.getName(), assetFile, digest); assets.put(assetId, asset); } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java new file mode 100644 index 000000000000..87b01a47b1c3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.asset; + +import org.apache.nifi.nar.FileDigestUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.HexFormat; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; + +/** + * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that + * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector. + * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can + * not interfere with the active configuration. + */ +public class StandardConnectorAssetManager implements AssetManager { + + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class); + + public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory"; + public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector_assets"; + + private volatile File assetStorageLocation; + private final Map assets = new ConcurrentHashMap<>(); + + @Override + public void initialize(final AssetManagerInitializationContext context) { + final String storageLocation = getStorageLocation(context); + + assetStorageLocation = new File(storageLocation); + if (!assetStorageLocation.exists()) { + try { + Files.createDirectories(assetStorageLocation.toPath()); + } catch (IOException e) { + throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created" + .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e); + } + } + + try { + recoverLocalAssets(); + } catch (final IOException e) { + throw new RuntimeException("Unable to access connector assets", e); + } + } + + @Override + public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException { + final String assetId = generateAssetId(); + final File assetFile = getFile(connectorId, assetId, assetName); + return saveAsset(connectorId, assetId, assetName, contents, assetFile); + } + + @Override + public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException { + final File assetFile = getFile(connectorId, assetId, assetName); + return saveAsset(connectorId, assetId, assetName, contents, assetFile); + } + + @Override + public Optional getAsset(final String id) { + return Optional.ofNullable(assets.get(id)); + } + + @Override + public List getAssets(final String connectorId) { + final List allAssets = new ArrayList<>(assets.values()); + final List ownerAssets = new ArrayList<>(); + for (final Asset asset : allAssets) { + if (asset.getOwnerIdentifier().equals(connectorId)) { + ownerAssets.add(asset); + } + } + return ownerAssets; + } + + @Override + public Asset createMissingAsset(final String connectorId, final String assetName) { + final String assetId = generateAssetId(); + final File file = getFile(connectorId, assetId, assetName); + final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null); + assets.put(assetId, asset); + return asset; + } + + @Override + public Optional deleteAsset(final String id) { + final Asset removed = assets.remove(id); + if (removed == null) { + return Optional.empty(); + } + + final File assetFile = removed.getFile(); + if (assetFile.exists()) { + deleteFile(assetFile); + + final File assetIdDir = assetFile.getParentFile(); + deleteFile(assetIdDir); + + final File connectorDir = assetIdDir.getParentFile(); + final File[] children = connectorDir.listFiles(); + if (children != null && children.length == 0) { + deleteFile(connectorDir); + } + } + + return Optional.of(removed); + } + + private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException { + final File dir = assetFile.getParentFile(); + if (!dir.exists()) { + try { + Files.createDirectories(dir.toPath()); + } catch (final IOException ioe) { + throw new IOException("Could not create directory in order to store connector asset", ioe); + } + } + + try { + Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING); + } catch (final Exception e) { + throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e); + } + + final String digest = computeDigest(assetFile); + final Asset asset = new StandardAsset(assetId, connectorId, assetName, assetFile, digest); + assets.put(assetId, asset); + return asset; + } + + private void deleteFile(final File file) { + try { + Files.delete(file.toPath()); + } catch (final IOException e) { + logger.warn("Failed to delete [{}]", file.getAbsolutePath(), e); + } + } + + private String generateAssetId() { + return UUID.randomUUID().toString(); + } + + private File getFile(final String connectorId, final String assetId, final String assetName) { + final Path parentPath = assetStorageLocation.toPath().normalize(); + final Path assetPath = Paths.get(connectorId, assetId, assetName).normalize(); + final Path fullPath = parentPath.resolve(assetPath); + return fullPath.toFile(); + } + + private String getStorageLocation(final AssetManagerInitializationContext initializationContext) { + final String storageLocation = initializationContext.getProperties().get(ASSET_STORAGE_LOCATION_PROPERTY); + return storageLocation == null ? DEFAULT_ASSET_STORAGE_LOCATION : storageLocation; + } + + private void recoverLocalAssets() throws IOException { + final File[] connectorDirectories = assetStorageLocation.listFiles(); + if (connectorDirectories == null) { + throw new IOException("Unable to list files for connector asset storage location %s".formatted(assetStorageLocation.getAbsolutePath())); + } + + for (final File connectorDirectory : connectorDirectories) { + if (!connectorDirectory.isDirectory()) { + continue; + } + + final String connectorId = connectorDirectory.getName(); + final File[] assetIdDirectories = connectorDirectory.listFiles(); + if (assetIdDirectories == null) { + logger.warn("Unable to list directory [{}]", connectorDirectory.getAbsolutePath()); + continue; + } + + for (final File assetIdDirectory : assetIdDirectories) { + final String assetId = assetIdDirectory.getName(); + + final File[] assetFiles = assetIdDirectory.listFiles(); + if (assetFiles == null) { + logger.warn("Unable to list directory [{}]", assetIdDirectory.getAbsolutePath()); + continue; + } + + for (final File assetFile : assetFiles) { + final String assetName = assetFile.getName(); + final String digest = computeDigest(assetFile); + final Asset asset = new StandardAsset(assetId, connectorId, assetName, assetFile, digest); + assets.put(assetId, asset); + } + } + } + } + + private String computeDigest(final File file) throws IOException { + return HexFormat.of().formatHex(FileDigestUtils.getDigest(file)); + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/StandardConnectorValidationTrigger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/StandardConnectorValidationTrigger.java new file mode 100644 index 000000000000..2e92671d2dda --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/StandardConnectorValidationTrigger.java @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.time.Duration; +import java.util.Collections; +import java.util.HashSet; +import java.util.Set; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.function.BooleanSupplier; + +/** + * Standard implementation of ConnectorValidationTrigger that submits validation + * tasks to an ScheduledExecutorService for asynchronous execution. + */ +public class StandardConnectorValidationTrigger implements ConnectorValidationTrigger { + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorValidationTrigger.class); + + private final ScheduledExecutorService threadPool; + private final BooleanSupplier flowInitialized; + private final Set activelyValidating = Collections.synchronizedSet(new HashSet<>()); + + public StandardConnectorValidationTrigger(final ScheduledExecutorService threadPool, final BooleanSupplier flowInitialized) { + this.threadPool = threadPool; + this.flowInitialized = flowInitialized; + } + + @Override + public void triggerAsync(final ConnectorNode connector) { + // Avoid adding multiple validation tasks for the same connector concurrently. This is not 100% thread safe because when a task + // is rescheduled, there's a small window where a second thread could be scheduled after the Connector is removed from 'activelyValidating' and + // before the task is rescheduled. However, this is acceptable because having multiple threads validating concurrently is safe, it's just inefficient. + final boolean added = activelyValidating.add(connector); + if (!added) { + logger.debug("Connector {} is already undergoing validation; will not trigger another validation concurrently", connector); + return; + } + + if (!flowInitialized.getAsBoolean()) { + logger.debug("Triggered to perform validation on {} asynchronously but flow is not yet initialized so will ignore validation", connector); + reschedule(connector, Duration.ofSeconds(1)); + return; + } + + threadPool.submit(() -> { + try { + if (connector.isValidationPaused()) { + logger.debug("Connector {} is currently marked as having validation paused; will retry in 1 second", connector); + reschedule(connector, Duration.ofSeconds(1)); + return; + } + + trigger(connector); + + activelyValidating.remove(connector); + } catch (final Exception e) { + logger.error("Validation for connector {} failed; will retry in 5 seconds", connector, e); + reschedule(connector, Duration.ofSeconds(5)); + } + }); + } + + @Override + public void trigger(final ConnectorNode connector) { + connector.performValidation(); + } + + private void reschedule(final ConnectorNode connector, final Duration delay) { + activelyValidating.remove(connector); + threadPool.schedule(() -> triggerAsync(connector), delay.toMillis(), TimeUnit.MILLISECONDS); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/ParameterProviderSecretProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/ParameterProviderSecretProvider.java new file mode 100644 index 000000000000..c44d5f10fd22 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/ParameterProviderSecretProvider.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.controller.ParameterProviderNode; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterDescriptor; +import org.apache.nifi.parameter.ParameterGroup; + +import java.util.ArrayList; +import java.util.List; + +public class ParameterProviderSecretProvider implements SecretProvider { + private final ParameterProviderNode parameterProvider; + + public ParameterProviderSecretProvider(final ParameterProviderNode parameterProvider) { + this.parameterProvider = parameterProvider; + } + + @Override + public String getProviderId() { + return parameterProvider.getIdentifier(); + } + + @Override + public String getProviderName() { + return parameterProvider.getName(); + } + + @Override + public List getAllSecrets() { + final List secrets = new ArrayList<>(); + + final List parameterGroups = parameterProvider.fetchParameterValues(); + for (final ParameterGroup group : parameterGroups) { + for (final Parameter parameter : group.getParameters()) { + final Secret secret = createSecret(group.getGroupName(), parameter); + secrets.add(secret); + } + } + + return secrets; + } + + private Secret createSecret(final String groupName, final Parameter parameter) { + final ParameterDescriptor descriptor = parameter.getDescriptor(); + + return new StandardSecret.Builder() + .providerId(getProviderId()) + .providerName(getProviderName()) + .groupName(groupName) + .name(descriptor.getName()) + .fullyQualifiedName(getProviderName() + "." + groupName + "." + descriptor.getName()) + .description(descriptor.getDescription()) + .value(parameter.getValue()) + .authorizable(parameterProvider) + .build(); + } + + @Override + public List getSecrets(final List fullyQualifiedSecretNames) { + final List parameterGroups = parameterProvider.fetchParameterValues(fullyQualifiedSecretNames); + final List secrets = new ArrayList<>(); + for (final ParameterGroup group : parameterGroups) { + for (final Parameter parameter : group.getParameters()) { + final Secret secret = createSecret(group.getGroupName(), parameter); + secrets.add(secret); + } + } + + return secrets; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/ParameterProviderSecretsManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/ParameterProviderSecretsManager.java new file mode 100644 index 000000000000..524351a41370 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/ParameterProviderSecretsManager.java @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.controller.ParameterProviderNode; +import org.apache.nifi.controller.flow.FlowManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.function.Function; +import java.util.stream.Collectors; + +public class ParameterProviderSecretsManager implements SecretsManager { + private static final Logger logger = LoggerFactory.getLogger(ParameterProviderSecretsManager.class); + private FlowManager flowManager; + + @Override + public void initialize(final SecretsManagerInitializationContext initializationContext) { + this.flowManager = initializationContext.getFlowManager(); + } + + @Override + public List getAllSecrets() { + final List secrets = new ArrayList<>(); + for (final SecretProvider provider : getSecretProviders()) { + secrets.addAll(provider.getAllSecrets()); + } + + // Sort secrets by Provider Name, then Group Name, then Secret Name + secrets.sort(Comparator.comparing(Secret::getProviderName) + .thenComparing(Secret::getGroupName) + .thenComparing(Secret::getName)); + + return secrets; + } + + @Override + public Set getSecretProviders() { + final Set providers = new HashSet<>(); + for (final ParameterProviderNode parameterProviderNode : flowManager.getAllParameterProviders()) { + ValidationStatus validationStatus = parameterProviderNode.getValidationStatus(); + if (validationStatus != ValidationStatus.VALID) { + validationStatus = parameterProviderNode.performValidation(); + } + if (validationStatus != ValidationStatus.VALID) { + logger.debug("Will not use Parameter Provider {} as a Secret Provider because it is not valid", parameterProviderNode.getName()); + continue; + } + + providers.add(new ParameterProviderSecretProvider(parameterProviderNode)); + } + + return providers; + } + + @Override + public Optional getSecret(final SecretReference secretReference) { + final SecretProvider provider = findProvider(secretReference); + if (provider == null) { + return Optional.empty(); + } + + final List secrets = provider.getSecrets(List.of(secretReference.getFullyQualifiedName())); + if (secrets.isEmpty()) { + return Optional.empty(); + } + + return Optional.of(secrets.getFirst()); + } + + @Override + public Map getSecrets(final Set secretReferences) { + if (secretReferences.isEmpty()) { + return Map.of(); + } + + // Partition secret references by Provider + final Map> referencesByProvider = new HashMap<>(); + for (final SecretReference secretReference : secretReferences) { + final SecretProvider provider = findProvider(secretReference); + referencesByProvider.computeIfAbsent(provider, k -> new HashSet<>()).add(secretReference); + } + + final Map secrets = new HashMap<>(); + for (final Map.Entry> entry : referencesByProvider.entrySet()) { + final SecretProvider provider = entry.getKey(); + final Set references = entry.getValue(); + + // If no provider found, be sure to map to a null Secret rather than skipping + if (provider == null) { + for (final SecretReference secretReference : references) { + secrets.put(secretReference, null); + } + + continue; + } + + final List secretNames = new ArrayList<>(); + references.forEach(ref -> secretNames.add(ref.getFullyQualifiedName())); + final List retrievedSecrets = provider.getSecrets(secretNames); + final Map secretsByName = retrievedSecrets.stream() + .collect(Collectors.toMap(Secret::getFullyQualifiedName, Function.identity())); + + for (final SecretReference secretReference : references) { + final Secret secret = secretsByName.get(secretReference.getFullyQualifiedName()); + secrets.put(secretReference, secret); + } + } + + return secrets; + } + + private SecretProvider findProvider(final SecretReference secretReference) { + final Set providers = getSecretProviders(); + + // Search first by Provider ID, if it's provided. + final String providerId = secretReference.getProviderId(); + if (providerId != null) { + for (final SecretProvider provider : providers) { + if (providerId.equals(provider.getProviderId())) { + return provider; + } + } + + // If ID is provided but doesn't match, do not consider name. + return null; + } + + // No Provider found by ID, extract Provider Name so we can search by it. + // If not explicitly provided, extract from FQN, if it is provided. + String providerName = secretReference.getProviderName(); + if (providerName == null) { + final String fqn = secretReference.getFullyQualifiedName(); + if (fqn != null) { + final int dotIndex = fqn.indexOf('.'); + if (dotIndex > 0) { + providerName = fqn.substring(0, dotIndex); + } + } + } + + // Search by Provider Name + if (providerName != null) { + for (final SecretProvider provider : providers) { + if (providerName.equals(provider.getProviderName())) { + return provider; + } + } + } + + // No Provider found + return null; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/StandardSecret.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/StandardSecret.java new file mode 100644 index 000000000000..0a81955fe5bc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/StandardSecret.java @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.authorization.Resource; +import org.apache.nifi.authorization.resource.Authorizable; + +import java.util.Objects; + +public class StandardSecret implements AuthorizableSecret { + private final String providerId; + private final String providerName; + private final String groupName; + private final String name; + private final String fullyQualifiedName; + private final String description; + private final String value; + private final Authorizable authorizable; + + private StandardSecret(final Builder builder) { + this.providerId = builder.providerId; + this.providerName = builder.providerName; + this.groupName = builder.groupName; + this.name = builder.name; + this.description = builder.description; + this.value = builder.value; + this.authorizable = builder.authorizable; + this.fullyQualifiedName = builder.fullyQualifiedName == null ? groupName + "." + name : builder.fullyQualifiedName; + } + + @Override + public String getProviderId() { + return providerId; + } + + @Override + public String getProviderName() { + return providerName; + } + + @Override + public String getGroupName() { + return groupName; + } + + @Override + public String getName() { + return name; + } + + @Override + public String getDescription() { + return description; + } + + @Override + public String getValue() { + return value; + } + + @Override + public String getFullyQualifiedName() { + return fullyQualifiedName; + } + + @Override + public String toString() { + return "StandardSecret[providerName=%s, groupName=%s, name=%s]".formatted(providerName, groupName, name); + } + + @Override + public int hashCode() { + return Objects.hash(providerName, groupName, name); + } + + @Override + public boolean equals(final Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + final StandardSecret other = (StandardSecret) obj; + return Objects.equals(this.providerName, other.providerName) + && Objects.equals(this.groupName, other.groupName) + && Objects.equals(this.name, other.name); + } + + @Override + public Authorizable getParentAuthorizable() { + return authorizable.getParentAuthorizable(); + } + + @Override + public Resource getResource() { + return authorizable.getResource(); + } + + public static class Builder { + private String providerId; + private String providerName; + private String groupName; + private String name; + private String fullyQualifiedName; + private String description; + private String value; + private Authorizable authorizable; + + public Builder providerId(final String providerId) { + this.providerId = providerId; + return this; + } + + public Builder providerName(final String providerName) { + this.providerName = providerName; + return this; + } + + public Builder groupName(final String groupName) { + this.groupName = groupName; + return this; + } + + public Builder name(final String name) { + this.name = name; + return this; + } + + public Builder description(final String description) { + this.description = description; + return this; + } + + public Builder value(final String value) { + this.value = value; + return this; + } + + public Builder authorizable(final Authorizable authorizable) { + this.authorizable = authorizable; + return this; + } + + public Builder fullyQualifiedName(final String fullyQualifiedName) { + this.fullyQualifiedName = fullyQualifiedName; + return this; + } + + public StandardSecret build() { + if (groupName == null) { + throw new IllegalStateException("Group name is required"); + } + if (authorizable == null) { + throw new IllegalStateException("Authorizable is required"); + } + if (name == null || name.isEmpty()) { + throw new IllegalStateException("Secret name must be provided"); + } + if (fullyQualifiedName == null || fullyQualifiedName.isEmpty()) { + throw new IllegalStateException("Secret fully qualified name must be provided"); + } + if ((providerId == null || providerId.isEmpty()) && (providerName == null || providerName.isEmpty())) { + throw new IllegalStateException("Either Secret provider ID or provider name must be provided"); + } + + return new StandardSecret(this); + } + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/StandardSecretsManagerInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/StandardSecretsManagerInitializationContext.java new file mode 100644 index 000000000000..ae5013768a81 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/components/connector/secrets/StandardSecretsManagerInitializationContext.java @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.controller.flow.FlowManager; + +public class StandardSecretsManagerInitializationContext implements SecretsManagerInitializationContext { + private final FlowManager flowManager; + + public StandardSecretsManagerInitializationContext(final FlowManager flowManager) { + this.flowManager = flowManager; + } + + @Override + public FlowManager getFlowManager() { + return flowManager; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java index 82befdb40360..2268d7fa52a9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java @@ -16,6 +16,10 @@ */ package org.apache.nifi.controller; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; import org.apache.nifi.annotation.behavior.Restricted; @@ -43,13 +47,19 @@ import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ComponentState; +import org.apache.nifi.components.connector.components.ConnectorMethod; +import org.apache.nifi.components.connector.components.MethodArgument; import org.apache.nifi.components.validation.ValidationState; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.components.validation.ValidationTrigger; import org.apache.nifi.components.validation.VerifiableComponentFactory; import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.ConnectableFlowFileActivity; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Position; import org.apache.nifi.controller.exception.ProcessorInstantiationException; import org.apache.nifi.controller.scheduling.LifecycleState; @@ -136,6 +146,9 @@ public class StandardProcessorNode extends ProcessorNode implements Connectable { private static final Logger LOG = LoggerFactory.getLogger(StandardProcessorNode.class); + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) + .setDefaultPropertyInclusion(JsonInclude.Include.NON_NULL); public static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS; public static final String DEFAULT_YIELD_PERIOD = "1 sec"; @@ -180,6 +193,8 @@ public class StandardProcessorNode extends ProcessorNode implements Connectable private volatile BackoffMechanism backoffMechanism; private volatile String maxBackoffPeriod; + private final ConnectableFlowFileActivity flowFileActivity = new ConnectableFlowFileActivity(); + public StandardProcessorNode(final LoggableComponent processor, final String uuid, final ValidationContextFactory validationContextFactory, final ProcessScheduler scheduler, final ControllerServiceProvider controllerServiceProvider, final ReloadComponent reloadComponent, @@ -979,7 +994,9 @@ List getIncomingNonLoopConnections() { } @Override - public List verifyConfiguration(final ProcessContext context, final ComponentLog logger, final Map attributes, final ExtensionManager extensionManager) { + public List verifyConfiguration(final ProcessContext context, final ComponentLog logger, final Map attributes, final ExtensionManager extensionManager, + final ParameterLookup parameterLookup) { + final List results = new ArrayList<>(); try { @@ -987,7 +1004,7 @@ public List verifyConfiguration(final ProcessContext c final long startNanos = System.nanoTime(); // Call super's verifyConfig, which will perform component validation - results.addAll(super.verifyConfig(context.getProperties(), context.getAnnotationData(), getProcessGroup().getParameterContext())); + results.addAll(super.verifyConfig(context.getProperties(), context.getAnnotationData(), parameterLookup)); final long validationComplete = System.nanoTime(); // If any invalid outcomes from validation, we do not want to perform additional verification, because we only run additional verification when the component is valid. @@ -1121,89 +1138,88 @@ protected Collection computeValidationErrors(final ValidationC } @Override - public List validateConfig() { + public List validateConfig(final ValidationContext validationContext) { final List results = new ArrayList<>(); - final ParameterContext parameterContext = getParameterContext(); - if (parameterContext == null && !this.parameterReferences.isEmpty()) { - results.add(new ValidationResult.Builder() - .subject(RUN_SCHEDULE) - .input("Parameter Context") - .valid(false) - .explanation("Processor configuration references one or more Parameters but no Parameter Context is currently set on the Process Group.") - .build()); - } else { - for (final ParameterReference paramRef : parameterReferences) { - final Optional parameterRef = parameterContext.getParameter(paramRef.getParameterName()); - if (!parameterRef.isPresent()) { - results.add(new ValidationResult.Builder() - .subject(RUN_SCHEDULE) - .input(paramRef.getParameterName()) - .valid(false) - .explanation("Processor configuration references Parameter '" + paramRef.getParameterName() + - "' but the currently selected Parameter Context does not have a Parameter with that name") - .build()); - } else { - final ParameterDescriptor parameterDescriptor = parameterRef.get().getDescriptor(); - if (parameterDescriptor.isSensitive()) { - results.add(new ValidationResult.Builder() - .subject(RUN_SCHEDULE) - .input(parameterDescriptor.getName()) - .valid(false) - .explanation("Processor configuration cannot reference sensitive parameters") - .build()); - } - } - } - - final String schedulingPeriod = getSchedulingPeriod(); - final String evaluatedSchedulingPeriod = evaluateParameters(schedulingPeriod); + for (final ParameterReference paramRef : parameterReferences) { + final String paramName = paramRef.getParameterName(); - if (evaluatedSchedulingPeriod != null) { - switch (schedulingStrategy) { - case CRON_DRIVEN: { - try { - CronExpression.parse(evaluatedSchedulingPeriod); - } catch (final Exception e) { + if (!validationContext.isParameterDefined(paramName)) { + results.add(new ValidationResult.Builder() + .subject(RUN_SCHEDULE) + .input(paramName) + .valid(false) + .explanation("Processor configuration references Parameter '" + paramName + + "' but the currently selected Parameter Context does not have a Parameter with that name") + .build()); + } else { + final ParameterContext parameterContext = getParameterContext(); + if (parameterContext != null) { + final Optional parameterFromContext = parameterContext.getParameter(paramName); + if (parameterFromContext.isPresent()) { + final ParameterDescriptor parameterDescriptor = parameterFromContext.get().getDescriptor(); + if (parameterDescriptor.isSensitive()) { results.add(new ValidationResult.Builder() .subject(RUN_SCHEDULE) - .input(schedulingPeriod) + .input(parameterDescriptor.getName()) .valid(false) - .explanation("Scheduling Period is not a valid cron expression") + .explanation("Processor configuration cannot reference sensitive parameters") .build()); } break; } - case TIMER_DRIVEN: { - try { - final long schedulingNanos = FormatUtils.getTimeDuration(Objects.requireNonNull(evaluatedSchedulingPeriod), - TimeUnit.NANOSECONDS); - - if (schedulingNanos < 0) { - results.add(new ValidationResult.Builder() - .subject(RUN_SCHEDULE) - .input(schedulingPeriod) - .valid(false) - .explanation("Scheduling Period must be positive") - .build()); - } + } + } + } + + final String schedulingPeriod = getSchedulingPeriod(); + final String evaluatedSchedulingPeriod = validationContext.evaluateParameters(schedulingPeriod); - this.schedulingNanos.set(Math.max(MINIMUM_SCHEDULING_NANOS, schedulingNanos)); + if (evaluatedSchedulingPeriod != null) { + switch (schedulingStrategy) { + case CRON_DRIVEN: { + try { + CronExpression.parse(evaluatedSchedulingPeriod); + } catch (final Exception e) { + results.add(new ValidationResult.Builder() + .subject(RUN_SCHEDULE) + .input(schedulingPeriod) + .valid(false) + .explanation("Scheduling Period is not a valid cron expression") + .build()); + } + } + break; + case TIMER_DRIVEN: { + try { + final long schedulingNanos = FormatUtils.getTimeDuration(Objects.requireNonNull(evaluatedSchedulingPeriod), + TimeUnit.NANOSECONDS); - } catch (final Exception e) { + if (schedulingNanos < 0) { results.add(new ValidationResult.Builder() .subject(RUN_SCHEDULE) .input(schedulingPeriod) .valid(false) - .explanation("Scheduling Period is not a valid time duration") + .explanation("Scheduling Period must be positive") .build()); } - break; + + this.schedulingNanos.set(Math.max(MINIMUM_SCHEDULING_NANOS, schedulingNanos)); + + } catch (final Exception e) { + results.add(new ValidationResult.Builder() + .subject(RUN_SCHEDULE) + .input(schedulingPeriod) + .valid(false) + .explanation("Scheduling Period is not a valid time duration") + .build()); } } + break; } } + return results; } @@ -1634,6 +1650,9 @@ private void initiateStart(final ScheduledExecutorService taskScheduler, final l // Completion Timestamp is set to MAX_VALUE because we don't want to timeout until the task has a chance to run. final AtomicLong completionTimestampRef = new AtomicLong(Long.MAX_VALUE); + // Mark current time as latest activity time so that we don't show as idle when the processor was stopped. + flowFileActivity.reset(); + // Create a task to invoke the @OnScheduled annotation of the processor final Callable startupTask = () -> { final ScheduledState currentScheduleState = scheduledState.get(); @@ -1657,10 +1676,10 @@ private void initiateStart(final ScheduledExecutorService taskScheduler, final l LOG.debug("Cannot start {} because Processor is currently not valid; will try again after 5 seconds", StandardProcessorNode.this); - startupAttemptCount.incrementAndGet(); - if (startupAttemptCount.get() == 240 || startupAttemptCount.get() % 7200 == 0) { + final long attempt = startupAttemptCount.getAndIncrement(); + if (attempt % 7200 == 0) { final ValidationState validationState = getValidationState(); - procLog.error("Encountering difficulty starting. (Validation State is {}: {}). Will continue trying to start.", + procLog.warn("Encountering difficulty starting. (Validation State is {}: {}). Will continue trying to start.", validationState, validationState.getValidationErrors()); } @@ -1937,6 +1956,99 @@ private void triggerLifecycleMethod(final ProcessContext processContext, final C } } + @Override + public List getConnectorMethods() { + return getConnectorMethods(getProcessor().getClass()); + } + + @Override + public String invokeConnectorMethod(final String methodName, final Map jsonArguments, final ProcessContext processContext) throws InvocationFailedException { + final ConfigurableComponent component = getComponent(); + + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(getExtensionManager(), component.getClass(), getIdentifier())) { + final Method implementationMethod = discoverConnectorMethod(component.getClass(), methodName); + final MethodArgument[] methodArguments = getConnectorMethodArguments(methodName, implementationMethod, component); + final List argumentValues = new ArrayList<>(); + + for (final MethodArgument methodArgument : methodArguments) { + if (ProcessContext.class.equals(methodArgument.type())) { + continue; + } + + final String jsonValue = jsonArguments.get(methodArgument.name()); + if (jsonValue == null && methodArgument.required()) { + throw new IllegalArgumentException("Cannot invoke Connector Method '" + methodName + "' on " + this + " because the required argument '" + + methodArgument.name() + "' was not provided"); + } + + if (jsonValue == null) { + argumentValues.add(null); + } else { + try { + final Object argumentValue = OBJECT_MAPPER.readValue(jsonValue, methodArgument.type()); + argumentValues.add(argumentValue); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to deserialize argument '" + methodArgument.name() + "' as type " + methodArgument.type().getName() + + " for Connector Method '" + methodName + "' on " + this, e); + } + } + } + + // Inject ProcessContext if the method signature supports it + final Class[] argumentTypes = implementationMethod.getParameterTypes(); + if (argumentTypes.length > 0 && ProcessContext.class.isAssignableFrom(argumentTypes[0])) { + argumentValues.addFirst(processContext); + } + if (argumentTypes.length > 1 && ProcessContext.class.isAssignableFrom(argumentTypes[argumentTypes.length - 1])) { + argumentValues.add(processContext); + } + + try { + implementationMethod.setAccessible(true); + final Object result = implementationMethod.invoke(component, argumentValues.toArray()); + if (result == null) { + return null; + } + + return OBJECT_MAPPER.writeValueAsString(result); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to serialize return value for Connector Method '" + methodName + "' on " + this, e); + } catch (final Exception e) { + throw new InvocationFailedException(e); + } + } + } + + private MethodArgument[] getConnectorMethodArguments(final String methodName, final Method implementationMethod, final ConfigurableComponent component) throws InvocationFailedException { + if (implementationMethod == null) { + throw new InvocationFailedException("No such connector method '" + methodName + "' exists for " + component.getClass().getName()); + } + + final ConnectorMethod connectorMethodDefinition = implementationMethod.getAnnotation(ConnectorMethod.class); + final ComponentState[] componentStates = connectorMethodDefinition.allowedStates(); + final ComponentState currentState = getComponentState(); + final boolean validState = Set.of(componentStates).contains(currentState); + if (!validState) { + throw new IllegalStateException("Cannot invoke Connector Method '" + methodName + "' on " + this + " because Processor is in state " + currentState + + " but the Connector Method does not allow invocation in this state"); + } + + final MethodArgument[] methodArguments = connectorMethodDefinition.arguments(); + return methodArguments; + } + + private ComponentState getComponentState() { + final ScheduledState scheduledState = getScheduledState(); + + return switch (scheduledState) { + case DISABLED -> ComponentState.PROCESSOR_DISABLED; + case STOPPED -> ComponentState.STOPPED; + case RUNNING, RUN_ONCE -> ComponentState.RUNNING; + case STARTING -> ComponentState.STARTING; + case STOPPING -> ComponentState.STOPPING; + }; + } + /** * Marks the processor as fully stopped, and completes any futures that are to be completed as a result */ @@ -2181,4 +2293,8 @@ private void updateControllerServiceReferences() { } } + @Override + public FlowFileActivity getFlowFileActivity() { + return flowFileActivity; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flow/AbstractFlowManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flow/AbstractFlowManager.java index 88c55c1a9b18..98382d5f17ef 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flow/AbstractFlowManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flow/AbstractFlowManager.java @@ -57,11 +57,13 @@ import java.util.ArrayList; import java.util.Collections; +import java.util.HashMap; import java.util.HashSet; import java.util.LinkedHashMap; import java.util.LinkedHashSet; import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.Optional; import java.util.Set; import java.util.UUID; @@ -121,6 +123,21 @@ public ProcessGroup getGroup(final String id) { return allProcessGroups.get(requireNonNull(id)); } + @Override + public ProcessGroup getGroup(final String groupId, final String connectorId) { + final ProcessGroup group = allProcessGroups.get(requireNonNull(groupId)); + if (group == null) { + return null; + } + + // If we found the group, return it only if it has the correct connector ID + if (Objects.equals(group.getConnectorIdentifier().orElse(null), connectorId)) { + return group; + } + + return null; + } + @Override public void onProcessGroupAdded(final ProcessGroup group) { allProcessGroups.put(group.getIdentifier(), group); @@ -627,16 +644,27 @@ public ParameterContextManager getParameterContextManager() { @Override public ParameterContext createParameterContext(final String id, final String name, final String description, - final Map parameters, final List inheritedContextIds, - final ParameterProviderConfiguration parameterProviderConfiguration) { - final boolean namingConflict = parameterContextManager.getParameterContexts().stream() + final Map parameters, final List inheritedContextIds, + final ParameterProviderConfiguration parameterProviderConfiguration) { + + final ParameterReferenceManager referenceManager = new StandardParameterReferenceManager(this::getRootGroup); + return createParameterContext(id, name, description, parameters, inheritedContextIds, parameterProviderConfiguration, referenceManager, true); + } + + protected ParameterContext createParameterContext(final String id, final String name, final String description, + final Map parameters, final List inheritedContextIds, + final ParameterProviderConfiguration parameterProviderConfiguration, final ParameterReferenceManager referenceManager, + final boolean register) { + + if (register) { + final boolean namingConflict = parameterContextManager.getParameterContexts().stream() .anyMatch(paramContext -> paramContext.getName().equals(name)); - if (namingConflict) { - throw new IllegalStateException("Cannot create Parameter Context with name '" + name + "' because a Parameter Context already exists with that name"); + if (namingConflict) { + throw new IllegalStateException("Cannot create Parameter Context with name '" + name + "' because a Parameter Context already exists with that name"); + } } - final ParameterReferenceManager referenceManager = new StandardParameterReferenceManager(this); final ParameterContext parameterContext = new StandardParameterContext.Builder() .id(id) .name(name) @@ -659,10 +687,23 @@ public ParameterContext createParameterContext(final String id, final String nam parameterContext.setInheritedParameterContexts(parameterContextList); } - parameterContextManager.addParameterContext(parameterContext); + if (register) { + parameterContextManager.addParameterContext(parameterContext); + } + return parameterContext; } + @Override + public ParameterContext createEmptyParameterContext(final String id, final String name, final String description, final ProcessGroup rootGroup) { + final Map parameterMap = new HashMap<>(); + final List inheritedContextIds = new ArrayList<>(); + + final ParameterReferenceManager parameterReferenceManager = new StandardParameterReferenceManager(() -> rootGroup); + return createParameterContext(id, name, description, + parameterMap, inheritedContextIds, null, parameterReferenceManager, false); + } + @Override public void withParameterContextResolution(final Runnable parameterContextAction) { withParameterContextResolution.set(true); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flowanalysis/FlowAnalysisUtil.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flowanalysis/FlowAnalysisUtil.java index 7c8e8963b820..6e6c58bd9287 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flowanalysis/FlowAnalysisUtil.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/flowanalysis/FlowAnalysisUtil.java @@ -19,13 +19,13 @@ import org.apache.nifi.nar.ExtensionManager; import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; public class FlowAnalysisUtil { public static final String ENCRYPTED_SENSITIVE_VALUE_SUBSTITUTE = "*****"; - public static NiFiRegistryFlowMapper createMapper(ExtensionManager extensionManager) { + public static VersionedComponentFlowMapper createMapper(ExtensionManager extensionManager) { final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder() .mapPropertyDescriptors(true) .mapControllerServiceReferencesToVersionedId(true) @@ -36,7 +36,7 @@ public static NiFiRegistryFlowMapper createMapper(ExtensionManager extensionMana .mapAssetReferences(true) .build(); - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(extensionManager, flowMappingOptions) { + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(extensionManager, flowMappingOptions) { @Override public String getGroupId(String groupId) { return groupId; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java index 0e8b616ad7b3..256fa991a53c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java @@ -266,7 +266,7 @@ public void removeReference(final ParameterContext reference) { } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } @@ -279,16 +279,32 @@ public void verifyCanFetchParameters() { } @Override - public void fetchParameters() { + public List fetchParameterValues() { final ParameterProvider parameterProvider = parameterProviderRef.get().getParameterProvider(); final ConfigurationContext configurationContext = getConfigurationContext(); - List fetchedParameterGroups; + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(getExtensionManager(), parameterProvider.getClass(), parameterProvider.getIdentifier())) { - fetchedParameterGroups = parameterProvider.fetchParameters(configurationContext); + return parameterProvider.fetchParameters(configurationContext); } catch (final IOException | RuntimeException e) { throw new IllegalStateException(String.format("Error fetching parameters for %s: %s", this, e.getMessage()), e); } + } + + @Override + public List fetchParameterValues(final List fullyQualifiedParameterNames) { + final ParameterProvider parameterProvider = parameterProviderRef.get().getParameterProvider(); + final ConfigurationContext configurationContext = getConfigurationContext(); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(getExtensionManager(), parameterProvider.getClass(), parameterProvider.getIdentifier())) { + return parameterProvider.fetchParameters(configurationContext, fullyQualifiedParameterNames); + } catch (final IOException | RuntimeException e) { + throw new IllegalStateException(String.format("Error fetching parameters for %s: %s", this, e.getMessage()), e); + } + } + + @Override + public void fetchParameters() { + final List fetchedParameterGroups = fetchParameterValues(); if (fetchedParameterGroups == null || fetchedParameterGroups.isEmpty()) { return; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java index 29c0cf97aef0..9a9c66bc52d3 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java @@ -603,7 +603,10 @@ protected void commit(final Checkpoint checkpoint, final boolean asynchronous) { // Update the FlowFile Repository try { final Collection repoRecords = checkpoint.records.values(); - context.getFlowFileRepository().updateRepository((Collection) repoRecords); + if (!repoRecords.isEmpty()) { + context.getFlowFileRepository().updateRepository((Collection) repoRecords); + context.getConnectable().getFlowFileActivity().updateLatestActivityTime(); + } } catch (final IOException ioe) { // if we fail to commit the session, we need to roll back // the checkpoints as well because none of the checkpoints @@ -984,7 +987,7 @@ protected void updateProvenanceRepo(final Checkpoint checkpoint) { // Therefore, we create an Iterable that can iterate over each of these events, modifying them as needed, and returning them // in the appropriate order. This prevents an unnecessary step of creating an intermediate List and adding all of those values // to the List. - // This is done in a similar veign to how Java 8's streams work, iterating over the events and returning a processed version + // This is done in a similar vein to how Java 8's streams work, iterating over the events and returning a processed version // one-at-a-time as opposed to iterating over the entire Collection and putting the results in another Collection. However, // we don't want to change the Framework to require Java 8 at this time, because it's not yet as prevalent as we would desire final Map flowFileRecordMap = new HashMap<>(); @@ -993,6 +996,9 @@ protected void updateProvenanceRepo(final Checkpoint checkpoint) { flowFileRecordMap.put(flowFile.getAttribute(CoreAttributes.UUID.key()), flowFile); } + // Update connectable to indicate how much data was received and sent. + updateTransferCounts(recordsToSubmit); + final long commitNanos = System.nanoTime(); final List autoTermEvents = checkpoint.autoTerminatedEvents; final Iterable iterable = new Iterable<>() { @@ -1039,6 +1045,24 @@ public void remove() { provenanceRepo.registerEvents(iterable); } + private void updateTransferCounts(final Collection events) { + int createdCount = 0; + long createdBytes = 0L; + for (final ProvenanceEventRecord event : events) { + if (event.getEventType() == ProvenanceEventType.CREATE) { + createdCount++; + createdBytes += event.getFileSize(); + } + } + + context.getConnectable().getFlowFileActivity().updateTransferCounts( + provenanceReporter.getFlowFilesReceived() + createdCount, + // For bytes, consider received and fetched; we don't include fetched in count, as the FlowFile count didn't change. + provenanceReporter.getBytesReceived() + provenanceReporter.getBytesFetched() + createdBytes, + provenanceReporter.getFlowFilesSent(), + provenanceReporter.getBytesSent()); + } + private void updateEventContentClaims(final ProvenanceEventBuilder builder, final FlowFile flowFile, final StandardRepositoryRecord repoRecord) { final ContentClaim originalClaim = repoRecord.getOriginalClaim(); if (originalClaim == null) { @@ -1246,6 +1270,10 @@ protected synchronized void rollback(final boolean penalize, final boolean rollb resetWriteClaims(); resetReadClaim(); + if (!recordValues.isEmpty()) { + context.getConnectable().getFlowFileActivity().updateLatestActivityTime(); + } + if (recordsToHandle.isEmpty()) { LOG.trace("{} was rolled back, but no events were performed by this ProcessSession", this); acknowledgeRecords(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceInvocationHandler.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceInvocationHandler.java index 9b87040540c8..3cb82157cae2 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceInvocationHandler.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceInvocationHandler.java @@ -17,6 +17,8 @@ package org.apache.nifi.controller.service; import org.apache.commons.lang3.ClassUtils; +import org.apache.nifi.components.connector.components.ComponentState; +import org.apache.nifi.components.connector.components.ConnectorMethod; import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.ControllerServiceProxyWrapper; import org.apache.nifi.nar.ExtensionManager; @@ -29,7 +31,6 @@ import java.lang.reflect.Method; import java.lang.reflect.Proxy; import java.util.Arrays; -import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Set; @@ -38,15 +39,7 @@ public class StandardControllerServiceInvocationHandler implements ControllerServiceInvocationHandler { private static final Logger logger = LoggerFactory.getLogger(StandardControllerServiceInvocationHandler.class); private static final Method PROXY_WRAPPER_GET_WRAPPED_METHOD; - - private static final Set validDisabledMethods; static { - // methods that are okay to be called when the service is disabled. - final Set validMethods = new HashSet<>(); - validMethods.addAll(Arrays.asList(ControllerService.class.getMethods())); - validMethods.addAll(Arrays.asList(Object.class.getMethods())); - validDisabledMethods = Collections.unmodifiableSet(validMethods); - try { PROXY_WRAPPER_GET_WRAPPED_METHOD = ControllerServiceProxyWrapper.class.getMethod("getWrapped"); } catch (final NoSuchMethodException e) { @@ -54,6 +47,7 @@ public class StandardControllerServiceInvocationHandler implements ControllerSer } } + private final Set validDisabledMethods; private final ControllerService originalService; private final AtomicReference serviceNodeHolder = new AtomicReference<>(null); private final ExtensionManager extensionManager; @@ -73,6 +67,34 @@ public StandardControllerServiceInvocationHandler(final ExtensionManager extensi this.extensionManager = extensionManager; this.originalService = originalService; this.serviceNodeHolder.set(serviceNode); + validDisabledMethods = determineDisabledAllowedMethods(originalService); + } + + private Set determineDisabledAllowedMethods(final ControllerService implementation) { + // methods that are okay to be called when the service is disabled. + final Set validMethods = new HashSet<>(); + validMethods.addAll(Arrays.asList(ControllerService.class.getMethods())); + validMethods.addAll(Arrays.asList(Object.class.getMethods())); + + // If there are any Connector Methods that are allowed to be called when the service is STOPPED or STOPPING, add those as well. + // This allows us to perform actions from Connectors, such as obtaining a list of available resources, during Connector configuration, etc. + // while the service is currently stopped. + for (final Method method : implementation.getClass().getDeclaredMethods()) { + final ConnectorMethod connectorMethod = method.getAnnotation(ConnectorMethod.class); + if (connectorMethod == null) { + continue; + } + + final ComponentState[] allowedStates = connectorMethod.allowedStates(); + for (final ComponentState allowedState : allowedStates) { + if (allowedState == ComponentState.STOPPED || allowedState == ComponentState.STOPPING) { + validMethods.add(method); + break; + } + } + } + + return validMethods; } @Override diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java index bc16e2cdb33d..ac5d12ef9f7e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java @@ -16,6 +16,10 @@ */ package org.apache.nifi.controller.service; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; import org.apache.commons.lang3.StringUtils; import org.apache.nifi.annotation.behavior.Restricted; import org.apache.nifi.annotation.documentation.DeprecationNotice; @@ -36,6 +40,10 @@ import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ComponentState; +import org.apache.nifi.components.connector.components.ConnectorMethod; +import org.apache.nifi.components.connector.components.MethodArgument; import org.apache.nifi.components.validation.ValidationState; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.components.validation.ValidationTrigger; @@ -102,6 +110,9 @@ public class StandardControllerServiceNode extends AbstractComponentNode implements ControllerServiceNode { private static final Logger LOG = LoggerFactory.getLogger(StandardControllerServiceNode.class); + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) + .setDefaultPropertyInclusion(JsonInclude.Include.NON_NULL); private static final long INCREMENTAL_VALIDATION_DELAY_MS = 1000; private static final Duration MAXIMUM_DELAY = Duration.ofMinutes(10); @@ -507,7 +518,7 @@ public void verifyCanPerformVerification() { @Override public List verifyConfiguration(final ConfigurationContext context, final ComponentLog logger, final Map variables, - final ExtensionManager extensionManager) { + final ExtensionManager extensionManager, final ParameterLookup parameterLookup) { final List results = new ArrayList<>(); @@ -516,7 +527,7 @@ public List verifyConfiguration(final ConfigurationCon final long startNanos = System.nanoTime(); // Call super's verifyConfig, which will perform component validation - results.addAll(super.verifyConfig(context.getProperties(), context.getAnnotationData(), getProcessGroup() == null ? null : getProcessGroup().getParameterContext())); + results.addAll(super.verifyConfig(context.getProperties(), context.getAnnotationData(), parameterLookup)); final long validationComplete = System.nanoTime(); // If any invalid outcomes from validation, we do not want to perform additional verification, because we only run additional verification when the component is valid. @@ -607,7 +618,7 @@ public ValidationState performValidation(final ValidationContext validationConte } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } @@ -632,6 +643,13 @@ protected List validateConfig() { */ @Override public CompletableFuture enable(final ScheduledExecutorService scheduler, final long administrativeYieldMillis, final boolean completeExceptionallyOnFailure) { + return enable(scheduler, administrativeYieldMillis, completeExceptionallyOnFailure, null); + } + + @Override + public CompletableFuture enable(final ScheduledExecutorService scheduler, final long administrativeYieldMillis, final boolean completeExceptionallyOnFailure, + final ConfigurationContext providedConfigurationContext) { + final CompletableFuture future = new CompletableFuture<>(); if (!stateTransition.transitionToEnabling(ControllerServiceState.DISABLED, future)) { @@ -650,7 +668,9 @@ public CompletableFuture enable(final ScheduledExecutorService scheduler, scheduler.execute(new Runnable() { @Override public void run() { - final ConfigurationContext configContext = new StandardConfigurationContext(serviceNode, controllerServiceProvider, null); + final ConfigurationContext configContext = providedConfigurationContext == null + ? new StandardConfigurationContext(serviceNode, controllerServiceProvider, null) + : providedConfigurationContext; if (!isActive()) { LOG.warn("Enabling {} stopped: no active status", serviceNode); @@ -659,9 +679,17 @@ public void run() { return; } - // Perform Validation and evaluate status before continuing - performValidation(); - final ValidationState validationState = getValidationState(); + // Perform validation - if a ConfigurationContext was provided, validate against its properties + final ValidationState validationState; + if (providedConfigurationContext == null) { + performValidation(); + validationState = getValidationState(); + } else { + final Map properties = providedConfigurationContext.getAllProperties(); + final ValidationContext validationContext = createValidationContext(properties, getAnnotationData(), getParameterLookup(), true); + validationState = performValidation(validationContext); + } + final ValidationStatus validationStatus = validationState.getStatus(); if (validationStatus == ValidationStatus.VALID) { LOG.debug("Enabling {} proceeding after performing validation", serviceNode); @@ -685,6 +713,7 @@ public void run() { LOG.debug("Validation rescheduling rejected for {}", serviceNode, e); future.completeExceptionally(new IllegalStateException("Enabling %s rejected: Validation Status [%s] Errors %s".formatted(serviceNode, validationStatus, errors))); } + // Enable command rescheduled or rejected return; } @@ -761,6 +790,12 @@ public CompletableFuture disable(final ScheduledExecutorService scheduler) } final CompletableFuture future = new CompletableFuture<>(); + // If already disabled, complete immediately + if (getState() == ControllerServiceState.DISABLED) { + future.complete(null); + return future; + } + final boolean transitioned = this.stateTransition.transitionToDisabling(ControllerServiceState.ENABLING, future); if (transitioned) { // If we transitioned from ENABLING to DISABLING, we need to immediately complete the disable @@ -910,6 +945,98 @@ public void migrateConfiguration(final Map originalPropertyValue } } + @Override + public String invokeConnectorMethod(final String methodName, final Map jsonArguments, final ConfigurationContext configurationContext) throws InvocationFailedException { + final ConfigurableComponent component = getComponent(); + + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(getExtensionManager(), component.getClass(), getIdentifier())) { + final Method implementationMethod = discoverConnectorMethod(component.getClass(), methodName); + final MethodArgument[] methodArguments = getConnectorMethodArguments(methodName, implementationMethod, component); + final List argumentValues = new ArrayList<>(); + + for (final MethodArgument methodArgument : methodArguments) { + if (ConfigurationContext.class.equals(methodArgument.type())) { + continue; + } + + final String jsonValue = jsonArguments.get(methodArgument.name()); + if (jsonValue == null && methodArgument.required()) { + throw new IllegalArgumentException("Cannot invoke Connector Method '" + methodName + "' on " + this + " because the required argument '" + + methodArgument.name() + "' was not provided"); + } + + if (jsonValue == null) { + argumentValues.add(null); + } else { + try { + final Object argumentValue = OBJECT_MAPPER.readValue(jsonValue, methodArgument.type()); + argumentValues.add(argumentValue); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to deserialize argument '" + methodArgument.name() + "' as type " + methodArgument.type().getName() + + " for Connector Method '" + methodName + "' on " + this, e); + } + } + } + + // Inject ConfigurationContext if the method signature supports it + final Class[] argumentTypes = implementationMethod.getParameterTypes(); + if (argumentTypes.length > 0 && ConfigurationContext.class.isAssignableFrom(argumentTypes[0])) { + argumentValues.addFirst(configurationContext); + } + if (argumentTypes.length > 1 && ConfigurationContext.class.isAssignableFrom(argumentTypes[argumentTypes.length - 1])) { + argumentValues.add(configurationContext); + } + + try { + implementationMethod.setAccessible(true); + final Object result = implementationMethod.invoke(component, argumentValues.toArray()); + if (result == null) { + return null; + } + + return OBJECT_MAPPER.writeValueAsString(result); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to serialize return value for Connector Method '" + methodName + "' on " + this, e); + } catch (final Exception e) { + throw new InvocationFailedException(e); + } + } + } + + @Override + public List getConnectorMethods() { + return getConnectorMethods(getControllerServiceImplementation().getClass()); + } + + private MethodArgument[] getConnectorMethodArguments(final String methodName, final Method implementationMethod, final ConfigurableComponent component) throws InvocationFailedException { + if (implementationMethod == null) { + throw new InvocationFailedException("No such connector method '" + methodName + "' exists for " + component.getClass().getName()); + } + + final ConnectorMethod connectorMethodDefinition = implementationMethod.getAnnotation(ConnectorMethod.class); + final ComponentState[] componentStates = connectorMethodDefinition.allowedStates(); + final ComponentState currentState = getComponentState(); + final boolean validState = Set.of(componentStates).contains(currentState); + if (!validState) { + throw new IllegalStateException("Cannot invoke Connector Method '" + methodName + "' on " + this + " because Processor is in state " + currentState + + " but the Connector Method does not allow invocation in this state"); + } + + final MethodArgument[] methodArguments = connectorMethodDefinition.arguments(); + return methodArguments; + } + + private ComponentState getComponentState() { + final ControllerServiceState scheduledState = getState(); + + return switch (scheduledState) { + case DISABLED -> ComponentState.STOPPED; + case DISABLING -> ComponentState.STOPPING; + case ENABLING -> ComponentState.STARTING; + case ENABLED -> ComponentState.RUNNING; + }; + } + @Override protected void performFlowAnalysisOnThis() { getValidationContextFactory().getFlowAnalyzer().ifPresent(flowAnalyzer -> flowAnalyzer.analyzeControllerService(this)); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java index 243a3c36c927..baec4ac365e1 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java @@ -269,7 +269,6 @@ public CompletableFuture enableControllerService(final ControllerServiceNo } serviceNode.verifyCanEnable(); - serviceNode.reloadAdditionalResourcesIfNecessary(); return processScheduler.enableControllerService(serviceNode); } @@ -320,7 +319,7 @@ private Collection removeControllerServicesWithUnavailabl } @Override - public Future enableControllerServicesAsync(final Collection serviceNodes) { + public CompletableFuture enableControllerServicesAsync(final Collection serviceNodes) { final CompletableFuture future = new CompletableFuture<>(); processScheduler.submitFrameworkTask(() -> { try { @@ -392,7 +391,7 @@ private void enableControllerServices(final Collection se } @Override - public Future enableControllerServiceAndDependencies(final ControllerServiceNode serviceNode) { + public CompletableFuture enableControllerServiceAndDependencies(final ControllerServiceNode serviceNode) { if (serviceNode.isActive()) { logger.debug("Enabling of Controller Service {} triggered but service already enabled", serviceNode); return CompletableFuture.completedFuture(null); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceResolver.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceResolver.java index 7b165b8a6cbb..442d301fddc9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceResolver.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceResolver.java @@ -30,7 +30,7 @@ import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.registry.flow.FlowSnapshotContainer; import org.apache.nifi.registry.flow.RegisteredFlowSnapshot; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import java.util.ArrayList; import java.util.Collection; @@ -47,13 +47,13 @@ public class StandardControllerServiceResolver implements ControllerServiceResol private final Authorizer authorizer; private final FlowManager flowManager; - private final NiFiRegistryFlowMapper flowMapper; + private final VersionedComponentFlowMapper flowMapper; private final ControllerServiceProvider controllerServiceProvider; private final ControllerServiceApiLookup controllerServiceApiLookup; public StandardControllerServiceResolver(final Authorizer authorizer, final FlowManager flowManager, - final NiFiRegistryFlowMapper flowMapper, + final VersionedComponentFlowMapper flowMapper, final ControllerServiceProvider controllerServiceProvider, final ControllerServiceApiLookup controllerServiceApiLookup) { this.authorizer = authorizer; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/events/VolatileBulletinRepository.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/events/VolatileBulletinRepository.java index 7ea59751e8ca..3779f5b34452 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/events/VolatileBulletinRepository.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/events/VolatileBulletinRepository.java @@ -290,6 +290,7 @@ private String getBulletinStoreKey(final Bulletin bulletin) { case FLOW_ANALYSIS_RULE -> FLOW_ANALYSIS_RULE_BULLETIN_STORE_KEY; case PARAMETER_PROVIDER -> PARAMETER_PROVIDER_BULLETIN_STORE_KEY; case FLOW_REGISTRY_CLIENT -> FLOW_REGISTRY_CLIENT_STORE_KEY; + case CONNECTOR -> bulletin.getSourceId(); default -> bulletin.getGroupId(); }; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java index 4d9e6bbeaf97..b67f79e56bf4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java @@ -113,7 +113,7 @@ import org.apache.nifi.registry.flow.diff.StandardComparableDataFlow; import org.apache.nifi.registry.flow.diff.StandardFlowComparator; import org.apache.nifi.registry.flow.diff.StaticDifferenceDescriptor; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.remote.PublicPort; import org.apache.nifi.remote.RemoteGroupPort; import org.apache.nifi.remote.StandardRemoteProcessGroupPortDescriptor; @@ -312,7 +312,7 @@ public ComponentAdditions addVersionedComponentsToProcessGroup(final ProcessGrou @Override public void synchronize(final ProcessGroup group, final VersionedExternalFlow versionedExternalFlow, final FlowSynchronizationOptions options) { - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(context.getExtensionManager(), context.getFlowMappingOptions()); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(context.getExtensionManager(), context.getFlowMappingOptions()); final VersionedProcessGroup versionedGroup = mapper.mapProcessGroup(group, context.getControllerServiceProvider(), context.getFlowManager(), true); final ComparableDataFlow localFlow = new StandardComparableDataFlow("Currently Loaded Flow", versionedGroup); @@ -998,7 +998,7 @@ private void removeTemporaryFunnel(final ProcessGroup group) { private Map componentsById(final ProcessGroup group, final Function> retrieveComponents) { return retrieveComponents.apply(group).stream() .collect(Collectors.toMap(component -> component.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(component.getIdentifier())), Function.identity())); + VersionedComponentFlowMapper.generateVersionedComponentId(component.getIdentifier())), Function.identity())); } private Map componentsById(final ProcessGroup group, final Function> retrieveComponents, @@ -1006,7 +1006,7 @@ private Map componentsById(final ProcessGroup group, final Functi return retrieveComponents.apply(group).stream() .collect(Collectors.toMap(component -> retrieveVersionedComponentId.apply(component).orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(retrieveId.apply(component))), Function.identity())); + VersionedComponentFlowMapper.generateVersionedComponentId(retrieveId.apply(component))), Function.identity())); } private void synchronizeFunnels(final ProcessGroup group, final VersionedProcessGroup proposed, final Map funnelsByVersionedId) { @@ -1242,7 +1242,7 @@ public void verifyCanSynchronize(final ProcessGroup group, final VersionedProces final Map removedInputPortsByVersionId = new HashMap<>(); group.getInputPorts() .forEach(port -> removedInputPortsByVersionId.put(port.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(port.getIdentifier())), port)); + VersionedComponentFlowMapper.generateVersionedComponentId(port.getIdentifier())), port)); flowContents.getInputPorts().stream() .map(VersionedPort::getIdentifier) @@ -1261,7 +1261,7 @@ public void verifyCanSynchronize(final ProcessGroup group, final VersionedProces final Map removedOutputPortsByVersionId = new HashMap<>(); group.getOutputPorts() .forEach(port -> removedOutputPortsByVersionId.put(port.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(port.getIdentifier())), port)); + VersionedComponentFlowMapper.generateVersionedComponentId(port.getIdentifier())), port)); flowContents.getOutputPorts().stream() .map(VersionedPort::getIdentifier) @@ -1288,7 +1288,7 @@ private void verifyCanInstantiateProcessors(final ProcessGroup group, final Set< group.findAllProcessors() .forEach(proc -> proposedProcessors.remove(proc.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(proc.getIdentifier())))); + VersionedComponentFlowMapper.generateVersionedComponentId(proc.getIdentifier())))); for (final VersionedProcessor processorToAdd : proposedProcessors.values()) { final String processorToAddClass = processorToAdd.getType(); @@ -1319,7 +1319,7 @@ private void verifyCanInstantiateControllerServices(final ProcessGroup group, fi group.findAllControllerServices() .forEach(service -> proposedServices.remove(service.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(service.getIdentifier())))); + VersionedComponentFlowMapper.generateVersionedComponentId(service.getIdentifier())))); for (final VersionedControllerService serviceToAdd : proposedServices.values()) { final String serviceToAddClass = serviceToAdd.getType(); @@ -1348,7 +1348,7 @@ private void verifyCanInstantiateConnections(final ProcessGroup group, final Set group.findAllConnections() .forEach(conn -> proposedConnections.remove(conn.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(conn.getIdentifier())))); + VersionedComponentFlowMapper.generateVersionedComponentId(conn.getIdentifier())))); for (final VersionedConnection connectionToAdd : proposedConnections.values()) { if (connectionToAdd.getPrioritizers() != null) { @@ -1377,13 +1377,18 @@ private ProcessGroup addProcessGroup(final ProcessGroup destination, final Versi final Map versionedParameterContexts, final Map parameterProviderReferences, ProcessGroup topLevelGroup) throws ProcessorInstantiationException { final String id = componentIdGenerator.generateUuid(proposed.getIdentifier(), proposed.getInstanceIdentifier(), destination.getIdentifier()); - final ProcessGroup group = context.getFlowManager().createProcessGroup(id); + final String connectorId = destination.getConnectorIdentifier().orElse(null); + final ProcessGroup group = context.getFlowManager().createProcessGroup(id, connectorId); group.setVersionedComponentId(proposed.getIdentifier()); group.setParent(destination); group.setName(proposed.getName()); destination.addProcessGroup(group); + // Connectors will have a single parameter context so if we are creating a group set the context of the parent process group. + if (connectorId != null) { + group.setParameterContext(destination.getParameterContext()); + } synchronize(group, proposed, versionedParameterContexts, parameterProviderReferences, topLevelGroup, true); return group; @@ -1949,7 +1954,8 @@ public void synchronizeProcessGroupSettings(final ProcessGroup processGroup, fin final ProcessGroup groupToUpdate; if (processGroup == null) { final String groupId = synchronizationOptions.getComponentIdGenerator().generateUuid(proposed.getIdentifier(), proposed.getInstanceIdentifier(), parentGroup.getIdentifier()); - final ProcessGroup group = context.getFlowManager().createProcessGroup(groupId); + final String connectorId = parentGroup.getConnectorIdentifier().orElse(null); + final ProcessGroup group = context.getFlowManager().createProcessGroup(groupId, connectorId); group.setVersionedComponentId(proposed.getIdentifier()); group.setParent(parentGroup); group.setName(proposed.getName()); @@ -2139,7 +2145,7 @@ private void verifyNotInherited(final String parameterContextId) { } private void updateParameterContext(final ProcessGroup group, final VersionedProcessGroup proposed, final Map versionedParameterContexts, - final Map parameterProviderReferences, final ComponentIdGenerator componentIdGenerator) { + final Map parameterProviderReferences, final ComponentIdGenerator componentIdGenerator) { // Update the Parameter Context final ParameterContext currentParamContext = group.getParameterContext(); final String proposedParameterContextName = proposed.getParameterContextName(); @@ -2164,9 +2170,9 @@ private void updateParameterContext(final ProcessGroup group, final VersionedPro final ParameterContext contextByName = getParameterContextByName(versionedParameterContext.getName()); if (contextByName == null) { final String parameterContextId = componentIdGenerator.generateUuid(versionedParameterContext.getName(), - versionedParameterContext.getName(), versionedParameterContext.getName()); + versionedParameterContext.getName(), versionedParameterContext.getName()); selectedParameterContext = createParameterContext(versionedParameterContext, parameterContextId, versionedParameterContexts, - parameterProviderReferences, componentIdGenerator); + parameterProviderReferences, componentIdGenerator); } else { selectedParameterContext = contextByName; addMissingConfiguration(versionedParameterContext, selectedParameterContext, versionedParameterContexts, parameterProviderReferences, componentIdGenerator); @@ -3072,7 +3078,7 @@ private void updateProcessor(final ProcessorNode processor, final VersionedProce private String getServiceInstanceId(final String serviceVersionedComponentId, final ProcessGroup group) { for (final ControllerServiceNode serviceNode : group.getControllerServices(false)) { final String versionedId = serviceNode.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(serviceNode.getIdentifier())); + VersionedComponentFlowMapper.generateVersionedComponentId(serviceNode.getIdentifier())); if (versionedId.equals(serviceVersionedComponentId)) { return serviceNode.getIdentifier(); } @@ -3691,7 +3697,7 @@ private Connectable getConnectable(final ProcessGroup group, final ConnectableCo final String rpgId = connectableComponent.getGroupId(); final Optional rpgOption = group.getRemoteProcessGroups().stream() .filter(component -> rpgId.equals(component.getIdentifier()) || rpgId.equals(component.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(component.getIdentifier())))) + VersionedComponentFlowMapper.generateVersionedComponentId(component.getIdentifier())))) .findAny(); if (rpgOption.isEmpty()) { @@ -3717,7 +3723,7 @@ private Connectable getConnectable(final ProcessGroup group, final ConnectableCo final String rpgId = connectableComponent.getGroupId(); final Optional rpgOption = group.getRemoteProcessGroups().stream() .filter(component -> rpgId.equals(component.getIdentifier()) || rpgId.equals(component.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(component.getIdentifier())))) + VersionedComponentFlowMapper.generateVersionedComponentId(component.getIdentifier())))) .findAny(); if (rpgOption.isEmpty()) { @@ -3918,12 +3924,12 @@ private void updateFlowAnalysisRule(final FlowAnalysisRuleNode flowAnalysisRule, } private boolean matchesId(final T component, final String id) { - return id.equals(component.getIdentifier()) || id.equals(component.getVersionedComponentId().orElse(NiFiRegistryFlowMapper.generateVersionedComponentId(component.getIdentifier()))); + return id.equals(component.getIdentifier()) || id.equals(component.getVersionedComponentId().orElse(VersionedComponentFlowMapper.generateVersionedComponentId(component.getIdentifier()))); } private boolean matchesGroupId(final ProcessGroup group, final String groupId) { return groupId.equals(group.getIdentifier()) || group.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(group.getIdentifier())).equals(groupId); + VersionedComponentFlowMapper.generateVersionedComponentId(group.getIdentifier())).equals(groupId); } private void findAllProcessors(final Set processors, final Set childGroups, final Map map) { @@ -3975,7 +3981,7 @@ private void verifyCanRemoveMissingComponents(final ProcessGroup processGroup, f // match group's current connections to proposed connections to determine if they've been removed for (final Connection connection : processGroup.getConnections()) { final String versionedId = connection.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(connection.getIdentifier())); + VersionedComponentFlowMapper.generateVersionedComponentId(connection.getIdentifier())); final VersionedConnection proposedConnection = proposedConnectionsByVersionedId.get(versionedId); if (proposedConnection == null) { // connection doesn't exist in proposed connections, make sure it doesn't have any data in it @@ -3994,7 +4000,7 @@ private void verifyCanRemoveMissingComponents(final ProcessGroup processGroup, f // match current child groups to proposed child groups to determine if they've been removed for (final ProcessGroup childGroup : processGroup.getProcessGroups()) { final String versionedId = childGroup.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(childGroup.getIdentifier())); + VersionedComponentFlowMapper.generateVersionedComponentId(childGroup.getIdentifier())); final VersionedProcessGroup proposedChildGroup = proposedGroupsByVersionedId.get(versionedId); if (proposedChildGroup == null) { if (verifyConnectionRemoval) { @@ -4020,7 +4026,7 @@ private ControllerServiceNode getVersionedControllerService(final ProcessGroup g for (final ControllerServiceNode serviceNode : group.getControllerServices(false)) { final String serviceNodeVersionedComponentId = serviceNode.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(serviceNode.getIdentifier())); + VersionedComponentFlowMapper.generateVersionedComponentId(serviceNode.getIdentifier())); if (serviceNodeVersionedComponentId.equals(versionedComponentId)) { return serviceNode; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java index 1e6deb27ae08..1dda6c268432 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java @@ -34,11 +34,13 @@ import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Funnel; import org.apache.nifi.connectable.LocalPort; import org.apache.nifi.connectable.Port; import org.apache.nifi.connectable.Position; import org.apache.nifi.connectable.Positionable; +import org.apache.nifi.connectable.ProcessGroupFlowFileActivity; import org.apache.nifi.controller.ComponentNode; import org.apache.nifi.controller.ConfigurationContext; import org.apache.nifi.controller.ControllerService; @@ -105,7 +107,7 @@ import org.apache.nifi.registry.flow.diff.StandardFlowComparator; import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; import org.apache.nifi.remote.PublicPort; import org.apache.nifi.remote.RemoteGroupPort; @@ -138,6 +140,7 @@ import java.util.concurrent.Callable; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; @@ -170,6 +173,7 @@ public final class StandardProcessGroup implements ProcessGroup { private final AtomicReference versionedComponentId = new AtomicReference<>(); private final AtomicReference versionControlInfo = new AtomicReference<>(); private static final SecureRandom randomGenerator = new SecureRandom(); + private final String connectorId; private final ProcessScheduler scheduler; private final ControllerServiceProvider controllerServiceProvider; @@ -196,6 +200,8 @@ public final class StandardProcessGroup implements ProcessGroup { private volatile ExecutionEngine executionEngine = ExecutionEngine.INHERITED; private volatile int maxConcurrentTasks = 1; private volatile String statelessFlowTimeout = "1 min"; + private volatile Authorizable explicitParentAuthorizable; + private final FlowFileActivity flowFileActivity = new ProcessGroupFlowFileActivity(this); private FlowFileConcurrency flowFileConcurrency = FlowFileConcurrency.UNBOUNDED; private volatile FlowFileGate flowFileGate = new UnboundedFlowFileGate(); @@ -225,7 +231,7 @@ public StandardProcessGroup(final String id, final ControllerServiceProvider ser final StateManagerProvider stateManagerProvider, final FlowManager flowManager, final ReloadComponent reloadComponent, final NodeTypeProvider nodeTypeProvider, final NiFiProperties nifiProperties, final StatelessGroupNodeFactory statelessGroupNodeFactory, - final AssetManager assetManager) { + final AssetManager assetManager, final String connectorId) { this.id = id; this.controllerServiceProvider = serviceProvider; @@ -239,6 +245,7 @@ public StandardProcessGroup(final String id, final ControllerServiceProvider ser this.reloadComponent = reloadComponent; this.nodeTypeProvider = nodeTypeProvider; this.assetManager = assetManager; + this.connectorId = connectorId; name = new AtomicReference<>(); position = new AtomicReference<>(new Position(0D, 0D)); @@ -294,7 +301,7 @@ public void setParent(final ProcessGroup newParent) { @Override public Authorizable getParentAuthorizable() { - return getParent(); + return explicitParentAuthorizable == null ? getParent() : explicitParentAuthorizable; } @Override @@ -332,6 +339,11 @@ public void setName(final String name) { setLoggingAttributes(); } + @Override + public Optional getConnectorIdentifier() { + return Optional.ofNullable(connectorId); + } + @Override public void setPosition(final Position position) { this.position.set(position); @@ -1574,6 +1586,7 @@ private DropFlowFileStatus handleDropAllFlowFiles(String dropRequestId, Function aggregateDropFlowFileStatus.setState(null); AtomicBoolean processedAtLeastOne = new AtomicBoolean(false); + final List> completionFutures = new ArrayList<>(); connections.stream() .map(Connection::getFlowFileQueue) @@ -1581,10 +1594,21 @@ private DropFlowFileStatus handleDropAllFlowFiles(String dropRequestId, Function .forEach(additionalDropFlowFileStatus -> { aggregate(aggregateDropFlowFileStatus, additionalDropFlowFileStatus); processedAtLeastOne.set(true); + completionFutures.add(additionalDropFlowFileStatus.getCompletionFuture()); }); if (processedAtLeastOne.get()) { resultDropFlowFileStatus = aggregateDropFlowFileStatus; + + // When all individual drop requests complete, mark the aggregate as complete + CompletableFuture.allOf(completionFutures.toArray(new CompletableFuture[0])) + .whenComplete((result, throwable) -> { + if (throwable != null) { + aggregateDropFlowFileStatus.setState(DropFlowFileState.FAILURE, throwable.getMessage()); + } else { + aggregateDropFlowFileStatus.setState(DropFlowFileState.COMPLETE); + } + }); } else { resultDropFlowFileStatus = null; } @@ -1715,7 +1739,7 @@ public RemoteProcessGroup getRemoteProcessGroup(final String id) { } @Override - public Future startProcessor(final ProcessorNode processor, final boolean failIfStopping) { + public CompletableFuture startProcessor(final ProcessorNode processor, final boolean failIfStopping) { readLock.lock(); try { if (getProcessor(processor.getIdentifier()) == null) { @@ -1731,8 +1755,6 @@ public Future startProcessor(final ProcessorNode processor, final boolean return CompletableFuture.completedFuture(null); } - processor.reloadAdditionalResourcesIfNecessary(); - return scheduler.startProcessor(processor, failIfStopping); } finally { readLock.unlock(); @@ -2095,7 +2117,7 @@ public ProcessGroup findProcessGroup(final String id) { return this; } - final ProcessGroup group = flowManager.getGroup(id); + final ProcessGroup group = flowManager.getGroup(id, getConnectorIdentifier().orElse(null)); if (group == null) { return null; } @@ -3842,16 +3864,22 @@ public void updateFlow(final VersionedExternalFlow proposedSnapshot, final Strin final ComponentScheduler defaultComponentScheduler = new DefaultComponentScheduler(controllerServiceProvider, stateLookup); final ComponentScheduler retainExistingStateScheduler = new RetainExistingStateComponentScheduler(this, defaultComponentScheduler); - final FlowSynchronizationOptions synchronizationOptions = new FlowSynchronizationOptions.Builder() + final FlowSynchronizationOptions.Builder flowSynchronizationBuilder = new FlowSynchronizationOptions.Builder() .componentIdGenerator(idGenerator) .componentComparisonIdLookup(VersionedComponent::getIdentifier) .componentScheduler(retainExistingStateScheduler) .ignoreLocalModifications(!verifyNotDirty) .updateDescendantVersionedFlows(updateDescendantVersionedFlows) .updateGroupSettings(updateSettings) - .updateRpgUrls(false) - .propertyDecryptor(value -> null) - .build(); + .updateRpgUrls(false); + // Connectors should not have encrypted values copied from versioned flow. However we do need to decrypt parameter references. + if (getConnectorIdentifier().isPresent()) { + flowSynchronizationBuilder.propertyDecryptor(value -> value); + } else { + flowSynchronizationBuilder.propertyDecryptor(value -> null); + } + + final FlowSynchronizationOptions synchronizationOptions = flowSynchronizationBuilder.build(); final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder() .mapSensitiveConfiguration(false) @@ -3924,7 +3952,7 @@ public Set getAncestorServiceIds() { // because this allows us to find the Controller Service when doing a Flow Diff. ancestorServiceIds = parentGroup.getControllerServices(true).stream() .map(cs -> cs.getVersionedComponentId().orElse( - NiFiRegistryFlowMapper.generateVersionedComponentId(cs.getIdentifier()))) + VersionedComponentFlowMapper.generateVersionedComponentId(cs.getIdentifier()))) .collect(Collectors.toSet()); } @@ -3969,7 +3997,7 @@ private Set getModifications() { } try { - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(extensionManager); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(extensionManager); final VersionedProcessGroup versionedGroup = mapper.mapProcessGroup(this, controllerServiceProvider, flowManager, false); final ComparableDataFlow currentFlow = new StandardComparableDataFlow("Local Flow", versionedGroup); @@ -4544,6 +4572,48 @@ public String getStatelessFlowTimeout() { return statelessFlowTimeout; } + @Override + public FlowFileActivity getFlowFileActivity() { + return flowFileActivity; + } + + @Override + public void setExplicitParentAuthorizable(final Authorizable parent) { + this.explicitParentAuthorizable = parent; + } + + @Override + public CompletableFuture purge() { + final CompletableFuture purgeFuture = new CompletableFuture<>(); + + Thread.ofVirtual().name("Purge " + this).start(() -> { + try { + stopProcessing().get(); + controllerServiceProvider.disableControllerServicesAsync(getControllerServices(true)).get(); + purgeQueues(); + removeComponents(this); + + purgeFuture.complete(null); + } catch (final Throwable t) { + purgeFuture.completeExceptionally(t); + } + }); + + return purgeFuture; + } + + private void purgeQueues() throws ExecutionException, InterruptedException { + for (final Connection connection : getConnections()) { + final FlowFileQueue flowFileQueue = connection.getFlowFileQueue(); + if (flowFileQueue.isEmpty()) { + continue; + } + + final DropFlowFileStatus status = connection.getFlowFileQueue().dropFlowFiles("purge-queues-" + getIdentifier(), "Framework"); + status.getCompletionFuture().get(); + } + } + @Override public void setStatelessFlowTimeout(final String statelessFlowTimeout) { if (statelessFlowTimeout == null) { diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ConnectorLogObserver.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ConnectorLogObserver.java new file mode 100644 index 000000000000..d6eba91f989a --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ConnectorLogObserver.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.logging; + +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.events.BulletinFactory; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.reporting.Severity; + +public class ConnectorLogObserver implements LogObserver { + private static final String CATEGORY = "Log Message"; + + private final BulletinRepository bulletinRepository; + private final ConnectorNode connectorNode; + + public ConnectorLogObserver(final BulletinRepository bulletinRepository, final ConnectorNode connectorNode) { + this.bulletinRepository = bulletinRepository; + this.connectorNode = connectorNode; + } + + @Override + public void onLogMessage(final LogMessage message) { + // Map LogLevel.WARN to Severity.WARNING so that we are consistent with the Severity enumeration. Else, just use whatever + // the LogLevel is (INFO and ERROR map directly and all others we will just accept as they are). + final String bulletinLevel = (message.getLogLevel() == LogLevel.WARN) ? Severity.WARNING.name() : message.getLogLevel().toString(); + bulletinRepository.addBulletin(BulletinFactory.createBulletin(connectorNode, CATEGORY, bulletinLevel, message.getMessage())); + } + + @Override + public String getComponentDescription() { + return connectorNode.toString(); + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ProcessorLogObserver.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ProcessorLogObserver.java index 7209492ca31d..597b3c8c1094 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ProcessorLogObserver.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/logging/ProcessorLogObserver.java @@ -31,7 +31,7 @@ public class ProcessorLogObserver implements LogObserver { private final BulletinRepository bulletinRepository; private final ProcessorNode processorNode; - public ProcessorLogObserver(BulletinRepository bulletinRepository, ProcessorNode processorNode) { + public ProcessorLogObserver(final BulletinRepository bulletinRepository, final ProcessorNode processorNode) { this.bulletinRepository = bulletinRepository; this.processorNode = processorNode; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterContext.java index 05636a3a45fe..d5445d2a8c2e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterContext.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterContext.java @@ -46,6 +46,7 @@ import java.util.Map; import java.util.Objects; import java.util.Optional; +import java.util.Set; import java.util.Stack; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.locks.Lock; @@ -147,6 +148,7 @@ public void setParameters(final Map updatedParameters) { } finally { writeLock.unlock(); } + alertReferencingComponents(parameterUpdates); } @@ -172,21 +174,25 @@ private Map getProposedParameters(final Map parameterUpdates) { - if (!parameterUpdates.isEmpty()) { - logger.debug("Parameter Context {} was updated. {} parameters changed ({}). Notifying all affected components.", this, parameterUpdates.size(), parameterUpdates); - - for (final ProcessGroup processGroup : parameterReferenceManager.getProcessGroupsBound(this)) { - try { - processGroup.onParameterContextUpdated(parameterUpdates); - } catch (final Exception e) { - logger.error("Failed to notify {} that Parameter Context was updated", processGroup, e); - } + if (parameterUpdates.isEmpty()) { + logger.debug("{} updated. No parameters changed so no existing components are affected.", this); + return; + } + + logger.debug("Parameter Context {} was updated. {} parameters changed ({}). Notifying all affected components.", this, parameterUpdates.size(), parameterUpdates); + for (final ProcessGroup processGroup : getBoundProcessGroups()) { + try { + processGroup.onParameterContextUpdated(parameterUpdates); + } catch (final Exception e) { + logger.error("Failed to notify {} that Parameter Context was updated", processGroup, e); } - } else { - logger.debug("Parameter Context {} was updated. {} parameters changed ({}). No existing components are affected.", this, parameterUpdates.size(), parameterUpdates); } } + protected Set getBoundProcessGroups() { + return parameterReferenceManager.getProcessGroupsBound(this); + } + /** * Returns a map from parameter name to ParameterUpdate for any actual updates to parameters. * @param currentParameters The current parameters diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterReferenceManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterReferenceManager.java index 117144986288..1d082963aea1 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterReferenceManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterReferenceManager.java @@ -22,7 +22,6 @@ import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.ProcessorNode; import org.apache.nifi.controller.PropertyConfiguration; -import org.apache.nifi.controller.flow.FlowManager; import org.apache.nifi.controller.service.ControllerServiceNode; import org.apache.nifi.groups.ProcessGroup; @@ -33,14 +32,15 @@ import java.util.Map; import java.util.Set; import java.util.function.Function; +import java.util.function.Supplier; public class StandardParameterReferenceManager implements ParameterReferenceManager { - private final FlowManager flowManager; + private final Supplier rootGroupLookup; // TODO: Consider reworking this so that we don't have to recurse through all components all the time and instead // have a 'caching' impl that AbstractComponentNode.setProperties() adds to/subtracts from. - public StandardParameterReferenceManager(final FlowManager flowManager) { - this.flowManager = flowManager; + public StandardParameterReferenceManager(final Supplier rootGroupLookup) { + this.rootGroupLookup = rootGroupLookup; } @Override @@ -57,13 +57,13 @@ public Set getControllerServicesReferencing(final Paramet public List getReferencedControllerServiceData(final ParameterContext parameterContext, final String parameterName) { final List referencedControllerServiceData = new ArrayList<>(); + final ProcessGroup rootGroup = rootGroupLookup.get(); final String versionedServiceId = parameterContext.getParameter(parameterName) .map(Parameter::getValue) - .map(this.flowManager::getControllerServiceNode) + .map(serviceId -> rootGroup.findControllerService(serviceId, true, true)) .flatMap(VersionedComponent::getVersionedComponentId) .orElse(null); - final ProcessGroup rootGroup = flowManager.getRootGroup(); final List referencingGroups = rootGroup.findAllProcessGroups(group -> group.referencesParameterContext(parameterContext)); for (final ProcessGroup group : referencingGroups) { @@ -89,7 +89,7 @@ public List getReferencedControllerSer @Override public Set getProcessGroupsBound(final ParameterContext parameterContext) { - final ProcessGroup rootGroup = flowManager.getRootGroup(); + final ProcessGroup rootGroup = rootGroupLookup.get(); final List referencingGroups = rootGroup.findAllProcessGroups(group -> group.referencesParameterContext(parameterContext)); return new HashSet<>(referencingGroups); @@ -99,14 +99,13 @@ private Set getComponentsReferencing(final Paramete final Function> componentFunction) { final Set referencingComponents = new HashSet<>(); - final ProcessGroup rootGroup = flowManager.getRootGroup(); + final ProcessGroup rootGroup = rootGroupLookup.get(); final List referencingGroups = rootGroup.findAllProcessGroups(group -> group.referencesParameterContext(parameterContext)); for (final ProcessGroup group : referencingGroups) { for (final T componentNode : componentFunction.apply(group)) { if (componentNode.isReferencingParameter(parameterName)) { referencingComponents.add(componentNode); - continue; } } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContext.java index 5edcb7c02637..9ee15d3eff1c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContext.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContext.java @@ -36,9 +36,12 @@ import org.apache.nifi.controller.service.ControllerServiceState; import org.apache.nifi.expression.ExpressionLanguageCompiler; import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.parameter.ExpressionLanguageAgnosticParameterParser; import org.apache.nifi.parameter.Parameter; -import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterLookup; +import org.apache.nifi.parameter.ParameterParser; import org.apache.nifi.parameter.ParameterReference; +import org.apache.nifi.parameter.ParameterTokenList; import java.util.Collection; import java.util.Collections; @@ -59,7 +62,7 @@ public class StandardValidationContext implements ValidationContext { private final String annotationData; private final String groupId; private final String componentId; - private final ParameterContext parameterContext; + private final ParameterLookup parameterLookup; private final AtomicReference> effectiveValuesRef = new AtomicReference<>(); private final boolean validateConnections; @@ -69,7 +72,7 @@ public StandardValidationContext( final String annotationData, final String groupId, final String componentId, - final ParameterContext parameterContext, + final ParameterLookup parameterLookup, final boolean validateConnections) { this.controllerServiceProvider = controllerServiceProvider; @@ -77,7 +80,7 @@ public StandardValidationContext( this.annotationData = annotationData; this.groupId = groupId; this.componentId = componentId; - this.parameterContext = parameterContext; + this.parameterLookup = parameterLookup; this.validateConnections = validateConnections; preparedQueries = new HashMap<>(properties.size()); @@ -85,7 +88,7 @@ public StandardValidationContext( final PropertyDescriptor desc = entry.getKey(); final PropertyConfiguration configuration = entry.getValue(); - String value = (configuration == null) ? null : configuration.getEffectiveValue(parameterContext); + String value = (configuration == null) ? null : configuration.getEffectiveValue(parameterLookup); if (value == null) { value = desc.getDefaultValue(); } @@ -103,12 +106,12 @@ public StandardValidationContext( @Override public PropertyValue newPropertyValue(final String rawValue) { final ResourceContext resourceContext = new StandardResourceContext(new StandardResourceReferenceFactory(), null); - return new StandardPropertyValue(resourceContext, rawValue, controllerServiceProvider, parameterContext, Query.prepareWithParametersPreEvaluated(rawValue)); + return new StandardPropertyValue(resourceContext, rawValue, controllerServiceProvider, parameterLookup, Query.prepareWithParametersPreEvaluated(rawValue)); } @Override public ExpressionLanguageCompiler newExpressionLanguageCompiler() { - return new StandardExpressionLanguageCompiler(parameterContext); + return new StandardExpressionLanguageCompiler(parameterLookup); } @Override @@ -123,9 +126,9 @@ public ValidationContext getControllerServiceValidationContext(final ControllerS @Override public PropertyValue getProperty(final PropertyDescriptor property) { final PropertyConfiguration configuredValue = properties.get(property); - final String effectiveValue = configuredValue == null ? property.getDefaultValue() : configuredValue.getEffectiveValue(parameterContext); + final String effectiveValue = configuredValue == null ? property.getDefaultValue() : configuredValue.getEffectiveValue(parameterLookup); final ResourceContext resourceContext = new StandardResourceContext(new StandardResourceReferenceFactory(), property); - return new StandardPropertyValue(resourceContext, effectiveValue, controllerServiceProvider, parameterContext, preparedQueries.get(property)); + return new StandardPropertyValue(resourceContext, effectiveValue, controllerServiceProvider, parameterLookup, preparedQueries.get(property)); } @Override @@ -139,7 +142,7 @@ public Map getProperties() { for (final Map.Entry entry : this.properties.entrySet()) { final PropertyDescriptor descriptor = entry.getKey(); final PropertyConfiguration configuration = entry.getValue(); - final String value = configuration == null ? descriptor.getDefaultValue() : configuration.getEffectiveValue(parameterContext); + final String value = configuration == null ? descriptor.getDefaultValue() : configuration.getEffectiveValue(parameterLookup); valueMap.put(entry.getKey(), value); } @@ -215,20 +218,20 @@ public Collection getReferencedParameters(final String propertyName) { @Override public boolean isParameterDefined(final String parameterName) { - if (parameterContext == null) { + if (parameterLookup == null) { return false; } - return parameterContext.getParameter(parameterName).isPresent(); + return parameterLookup.getParameter(parameterName).isPresent(); } @Override public boolean isParameterSet(final String parameterName) { - if (parameterContext == null) { + if (parameterLookup == null) { return false; } - final Optional parameterOption = parameterContext.getParameter(parameterName); + final Optional parameterOption = parameterLookup.getParameter(parameterName); if (!parameterOption.isPresent()) { return false; } @@ -242,6 +245,13 @@ public boolean isValidateConnections() { return validateConnections; } + @Override + public String evaluateParameters(final String value) { + final ParameterParser parameterParser = new ExpressionLanguageAgnosticParameterParser(); + final ParameterTokenList parameterTokenList = parameterParser.parseTokens(value); + return parameterTokenList.substitute(parameterLookup); + } + @Override public String toString() { return "StandardValidationContext[componentId=" + componentId + ", properties=" + properties + "]"; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContextFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContextFactory.java index 00a7b6852a71..a1d8178f91a4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContextFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/processor/StandardValidationContextFactory.java @@ -22,7 +22,7 @@ import org.apache.nifi.controller.ValidationContextFactory; import org.apache.nifi.controller.flowanalysis.FlowAnalyzer; import org.apache.nifi.controller.service.ControllerServiceProvider; -import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterLookup; import org.apache.nifi.validation.RuleViolationsManager; import java.util.Map; @@ -50,8 +50,8 @@ public StandardValidationContextFactory( @Override public ValidationContext newValidationContext(final Map properties, final String annotationData, final String groupId, final String componentId, - final ParameterContext parameterContext, final boolean validateConnections) { - return new StandardValidationContext(serviceProvider, properties, annotationData, groupId, componentId, parameterContext, validateConnections); + final ParameterLookup parameterLookup, final boolean validateConnections) { + return new StandardValidationContext(serviceProvider, properties, annotationData, groupId, componentId, parameterLookup, validateConnections); } @Override diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java index db1949e06f49..2d3bb71a3e49 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java @@ -45,7 +45,7 @@ import org.apache.nifi.parameter.ParameterLookup; import org.apache.nifi.parameter.ParameterUpdate; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.validation.RuleViolation; import org.apache.nifi.validation.RuleViolationsManager; import org.slf4j.Logger; @@ -70,7 +70,7 @@ public final class FlowAnalyzingRegistryClientNode implements FlowRegistryClient private final FlowAnalyzer flowAnalyzer; private final RuleViolationsManager ruleViolationsManager; private final FlowManager flowManager; - private final NiFiRegistryFlowMapper flowMapper; + private final VersionedComponentFlowMapper flowMapper; public FlowAnalyzingRegistryClientNode( final FlowRegistryClientNode node, @@ -78,7 +78,7 @@ public FlowAnalyzingRegistryClientNode( final FlowAnalyzer flowAnalyzer, final RuleViolationsManager ruleViolationsManager, final FlowManager flowManager, - final NiFiRegistryFlowMapper flowMapper + final VersionedComponentFlowMapper flowMapper ) { this.node = Objects.requireNonNull(node); this.serviceProvider = Objects.requireNonNull(serviceProvider); @@ -112,7 +112,8 @@ public RegisteredFlowSnapshot registerFlowSnapshot( } private boolean analyzeProcessGroupToRegister(final VersionedProcessGroup snapshot) { - final InstantiatedVersionedProcessGroup nonVersionedProcessGroup = flowMapper.mapNonVersionedProcessGroup(flowManager.getGroup(snapshot.getInstanceIdentifier()), serviceProvider); + final ProcessGroup group = flowManager.getGroup(snapshot.getInstanceIdentifier(), null); + final InstantiatedVersionedProcessGroup nonVersionedProcessGroup = flowMapper.mapNonVersionedProcessGroup(group, serviceProvider); flowAnalyzer.analyzeProcessGroup(nonVersionedProcessGroup); final List ruleViolations = ruleViolationsManager.getRuleViolationsForGroup(snapshot.getInstanceIdentifier()).stream() @@ -179,6 +180,13 @@ public void verifyCanUpdateProperties(final Map properties) { node.verifyCanUpdateProperties(properties); } + @Override + public ValidationContext createValidationContext(final Map propertyValues, final String annotationData, + final ParameterLookup parameterLookup, final boolean validateConnections) { + + return node.createValidationContext(propertyValues, annotationData, parameterLookup, validateConnections); + } + @Override public Set getReferencedParameterNames() { return node.getReferencedParameterNames(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/StandardFlowRegistryClientNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/StandardFlowRegistryClientNode.java index 9eb895fcec14..e8e936df2708 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/StandardFlowRegistryClientNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/StandardFlowRegistryClientNode.java @@ -28,6 +28,7 @@ import org.apache.nifi.components.ConfigVerificationResult.Outcome; import org.apache.nifi.components.ConfigurableComponent; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.components.validation.ValidationTrigger; @@ -118,7 +119,7 @@ public String getProcessGroupIdentifier() { } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/NiFiRegistryFlowMapper.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java similarity index 92% rename from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/NiFiRegistryFlowMapper.java rename to nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java index 9f847a24649a..1d845265c9bc 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/NiFiRegistryFlowMapper.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java @@ -21,6 +21,16 @@ import org.apache.nifi.asset.Asset; import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConnectorConfiguration; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.NamedStepConfiguration; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.components.connector.StringLiteralValue; import org.apache.nifi.components.listen.ListenPortDefinition; import org.apache.nifi.components.resource.ResourceCardinality; import org.apache.nifi.components.resource.ResourceDefinition; @@ -53,7 +63,10 @@ import org.apache.nifi.flow.PortType; import org.apache.nifi.flow.Position; import org.apache.nifi.flow.VersionedAsset; +import org.apache.nifi.flow.VersionedConfigurationStep; import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedConnector; +import org.apache.nifi.flow.VersionedConnectorValueReference; import org.apache.nifi.flow.VersionedControllerService; import org.apache.nifi.flow.VersionedFlowAnalysisRule; import org.apache.nifi.flow.VersionedFlowCoordinates; @@ -108,7 +121,7 @@ import java.util.function.Supplier; import java.util.stream.Collectors; -public class NiFiRegistryFlowMapper { +public class VersionedComponentFlowMapper { private static final String ENCRYPTED_PREFIX = "enc{"; private static final String ENCRYPTED_SUFFIX = "}"; @@ -122,11 +135,11 @@ public class NiFiRegistryFlowMapper { // created before attempting to create the connection, where the ConnectableDTO is converted. private final Map versionedComponentIds = new HashMap<>(); - public NiFiRegistryFlowMapper(final ExtensionManager extensionManager) { + public VersionedComponentFlowMapper(final ExtensionManager extensionManager) { this(extensionManager, FlowMappingOptions.DEFAULT_OPTIONS); } - public NiFiRegistryFlowMapper(final ExtensionManager extensionManager, final FlowMappingOptions flowMappingOptions) { + public VersionedComponentFlowMapper(final ExtensionManager extensionManager, final FlowMappingOptions flowMappingOptions) { this.extensionManager = extensionManager; this.flowMappingOptions = flowMappingOptions; @@ -1003,6 +1016,7 @@ private VersionedParameter mapParameter(final Parameter parameter, final String .toList(); versionedParameter.setReferencedAssets(assetIds); } + return versionedParameter; } @@ -1018,4 +1032,79 @@ private org.apache.nifi.flow.ScheduledState mapScheduledState(final ScheduledSta ? org.apache.nifi.flow.ScheduledState.DISABLED : org.apache.nifi.flow.ScheduledState.ENABLED; } + + public VersionedConnector mapConnector(final ConnectorNode connectorNode) { + final VersionedConnector versionedConnector = new VersionedConnector(); + versionedConnector.setInstanceIdentifier(connectorNode.getIdentifier()); + versionedConnector.setName(connectorNode.getName()); + versionedConnector.setScheduledState(mapConnectorState(connectorNode.getDesiredState())); + versionedConnector.setType(connectorNode.getCanonicalClassName()); + versionedConnector.setBundle(mapBundle(connectorNode.getBundleCoordinate())); + + final List activeFlowConfiguration = createVersionedConfigurationSteps(connectorNode.getActiveFlowContext()); + versionedConnector.setActiveFlowConfiguration(activeFlowConfiguration); + + final List workingFlowConfiguration = createVersionedConfigurationSteps(connectorNode.getWorkingFlowContext()); + versionedConnector.setWorkingFlowConfiguration(workingFlowConfiguration); + + return versionedConnector; + } + + private List createVersionedConfigurationSteps(final FrameworkFlowContext flowContext) { + if (flowContext == null) { + return Collections.emptyList(); + } + + final ConnectorConfiguration configuration = flowContext.getConfigurationContext().toConnectorConfiguration(); + final List configurationSteps = new ArrayList<>(); + + for (final NamedStepConfiguration stepConfiguration : configuration.getNamedStepConfigurations()) { + final VersionedConfigurationStep versionedConfigurationStep = new VersionedConfigurationStep(); + versionedConfigurationStep.setName(stepConfiguration.stepName()); + versionedConfigurationStep.setProperties(mapPropertyValues(stepConfiguration.configuration())); + + configurationSteps.add(versionedConfigurationStep); + } + + return configurationSteps; + } + + private Map mapPropertyValues(final StepConfiguration configuration) { + final Map versionedProperties = new HashMap<>(); + for (final Map.Entry entry : configuration.getPropertyValues().entrySet()) { + final ConnectorValueReference valueReference = entry.getValue(); + if (valueReference == null) { + continue; + } + + final VersionedConnectorValueReference versionedReference = new VersionedConnectorValueReference(); + versionedReference.setValueType(valueReference.getValueType().name()); + + switch (valueReference) { + case StringLiteralValue stringLiteral -> versionedReference.setValue(stringLiteral.getValue()); + case AssetReference assetRef -> versionedReference.setAssetIds(assetRef.getAssetIdentifiers()); + case SecretReference secretRef -> { + versionedReference.setProviderId(secretRef.getProviderId()); + versionedReference.setProviderName(secretRef.getProviderName()); + versionedReference.setSecretName(secretRef.getSecretName()); + versionedReference.setFullyQualifiedSecretName(secretRef.getFullyQualifiedName()); + } + } + + versionedProperties.put(entry.getKey(), versionedReference); + } + + return versionedProperties; + } + + private org.apache.nifi.flow.ScheduledState mapConnectorState(final ConnectorState connectorState) { + if (connectorState == null) { + return null; + } + + return switch (connectorState) { + case RUNNING, STARTING -> org.apache.nifi.flow.ScheduledState.RUNNING; + default -> org.apache.nifi.flow.ScheduledState.ENABLED; + }; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/remote/StandardRemoteProcessGroup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/remote/StandardRemoteProcessGroup.java index edd34d4fac5e..7d63e2b66b9c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/remote/StandardRemoteProcessGroup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/remote/StandardRemoteProcessGroup.java @@ -67,7 +67,7 @@ import java.util.Optional; import java.util.Set; import java.util.UUID; -import java.util.concurrent.Future; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; @@ -1102,7 +1102,7 @@ public void startTransmitting(final RemoteGroupPort port) { } @Override - public Future stopTransmitting() { + public CompletableFuture stopTransmitting() { writeLock.lock(); try { verifyCanStopTransmitting(); @@ -1117,13 +1117,15 @@ public Future stopTransmitting() { configuredToTransmit.set(false); - return scheduler.submitFrameworkTask(this::waitForPortShutdown); + final CompletableFuture completableFuture = new CompletableFuture<>(); + scheduler.submitFrameworkTask(() -> waitForPortShutdown(completableFuture)); + return completableFuture; } finally { writeLock.unlock(); } } - private void waitForPortShutdown() { + private void waitForPortShutdown(final CompletableFuture completableFuture) { // Wait for the ports to stop try { for (final RemoteGroupPort port : getInputPorts()) { @@ -1149,6 +1151,7 @@ private void waitForPortShutdown() { } } finally { transmitting.set(false); + completableFuture.complete(null); } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager index 6fbf05d94972..3a438fb65dd7 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.asset.AssetManager @@ -13,4 +13,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -org.apache.nifi.asset.StandardAssetManager \ No newline at end of file +org.apache.nifi.asset.StandardAssetManager +org.apache.nifi.asset.StandardConnectorAssetManager \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager new file mode 100644 index 000000000000..082fffbd8f19 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/resources/META-INF/services/org.apache.nifi.components.connector.secrets.SecretsManager @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.components.connector.secrets.ParameterProviderSecretsManager diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/asset/StandardConnectorAssetManagerTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/asset/StandardConnectorAssetManagerTest.java new file mode 100644 index 000000000000..fa13d28bba35 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/asset/StandardConnectorAssetManagerTest.java @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.asset; + +import org.apache.nifi.controller.NodeTypeProvider; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.ByteArrayInputStream; +import java.io.File; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; + +class StandardConnectorAssetManagerTest { + + @Test + void testInitializeRecoversExistingAssets(@TempDir final Path tempDir) throws Exception { + final Path storageDirectory = tempDir.resolve("assets"); + final String connectorIdentifier = "connector-1"; + final String assetIdentifier = "asset-1"; + final String assetName = "asset.txt"; + + final Path assetDirectory = storageDirectory.resolve(connectorIdentifier).resolve(assetIdentifier); + Files.createDirectories(assetDirectory); + final Path assetPath = assetDirectory.resolve(assetName); + Files.writeString(assetPath, "existing-content", StandardCharsets.UTF_8); + + final StandardConnectorAssetManager manager = new StandardConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(storageDirectory.toFile()); + manager.initialize(context); + + final List assets = manager.getAssets(connectorIdentifier); + assertEquals(1, assets.size()); + + final Asset recoveredAsset = assets.getFirst(); + assertEquals(assetName, recoveredAsset.getName()); + assertEquals(connectorIdentifier, recoveredAsset.getOwnerIdentifier()); + assertTrue(recoveredAsset.getDigest().isPresent()); + assertEquals(assetPath.toFile().getAbsolutePath(), recoveredAsset.getFile().getAbsolutePath()); + } + + @Test + void testCreateAssetAndGetAsset(@TempDir final Path tempDir) throws Exception { + final Path storageDirectory = tempDir.resolve("assets"); + final StandardConnectorAssetManager manager = new StandardConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(storageDirectory.toFile()); + manager.initialize(context); + + assertTrue(Files.isDirectory(storageDirectory)); + + final String connectorIdentifier = "connector-1"; + final String assetName = "created-asset.txt"; + final byte[] contents = "created-content".getBytes(StandardCharsets.UTF_8); + final InputStream inputStream = new ByteArrayInputStream(contents); + + final Asset createdAsset = manager.createAsset(connectorIdentifier, assetName, inputStream); + assertNotNull(createdAsset); + assertEquals(connectorIdentifier, createdAsset.getOwnerIdentifier()); + assertEquals(assetName, createdAsset.getName()); + assertTrue(createdAsset.getDigest().isPresent()); + assertTrue(createdAsset.getFile().exists()); + + final Optional retrievedByIdentifier = manager.getAsset(createdAsset.getIdentifier()); + assertTrue(retrievedByIdentifier.isPresent()); + assertEquals(createdAsset.getIdentifier(), retrievedByIdentifier.get().getIdentifier()); + + final List connectorAssets = manager.getAssets(connectorIdentifier); + assertEquals(1, connectorAssets.size()); + assertEquals(createdAsset.getIdentifier(), connectorAssets.get(0).getIdentifier()); + } + + @Test + void testCreateMissingAssetDoesNotCreateFile(@TempDir final Path tempDir) { + final Path storageDirectory = tempDir.resolve("assets"); + final StandardConnectorAssetManager manager = new StandardConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(storageDirectory.toFile()); + manager.initialize(context); + + final String connectorIdentifier = "connector-2"; + final String assetName = "missing-asset.txt"; + + final Asset missingAsset = manager.createMissingAsset(connectorIdentifier, assetName); + assertNotNull(missingAsset); + assertEquals(connectorIdentifier, missingAsset.getOwnerIdentifier()); + assertEquals(assetName, missingAsset.getName()); + assertFalse(missingAsset.getDigest().isPresent()); + assertFalse(missingAsset.getFile().exists()); + + final Optional retrieved = manager.getAsset(missingAsset.getIdentifier()); + assertTrue(retrieved.isPresent()); + assertEquals(missingAsset.getIdentifier(), retrieved.get().getIdentifier()); + + final List connectorAssets = manager.getAssets(connectorIdentifier); + assertEquals(1, connectorAssets.size()); + assertEquals(missingAsset.getIdentifier(), connectorAssets.getFirst().getIdentifier()); + } + + @Test + void testDeleteAssetRemovesFilesAndDirectories(@TempDir final Path tempDir) throws Exception { + final Path storageDirectory = tempDir.resolve("assets"); + final StandardConnectorAssetManager manager = new StandardConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(storageDirectory.toFile()); + manager.initialize(context); + + final String connectorIdentifier = "connector-3"; + final String assetName = "deletable-asset.txt"; + final byte[] contents = "deletable-content".getBytes(StandardCharsets.UTF_8); + final InputStream inputStream = new ByteArrayInputStream(contents); + + final Asset asset = manager.createAsset(connectorIdentifier, assetName, inputStream); + final File assetFile = asset.getFile(); + final File assetDirectory = assetFile.getParentFile(); + final File connectorDirectory = assetDirectory.getParentFile(); + + assertTrue(assetFile.exists()); + assertTrue(assetDirectory.exists()); + assertTrue(connectorDirectory.exists()); + + final Optional deleted = manager.deleteAsset(asset.getIdentifier()); + assertTrue(deleted.isPresent()); + assertFalse(assetFile.exists()); + assertFalse(assetDirectory.exists()); + assertFalse(connectorDirectory.exists()); + + assertTrue(manager.getAsset(asset.getIdentifier()).isEmpty()); + assertTrue(manager.getAssets(connectorIdentifier).isEmpty()); + } + + @Test + void testDeleteNonExistentAssetReturnsEmpty(@TempDir final Path tempDir) { + final Path storageDirectory = tempDir.resolve("assets"); + final StandardConnectorAssetManager manager = new StandardConnectorAssetManager(); + final AssetManagerInitializationContext context = createInitializationContext(storageDirectory.toFile()); + manager.initialize(context); + + final Optional deleted = manager.deleteAsset("non-existent-id"); + assertTrue(deleted.isEmpty()); + } + + private AssetManagerInitializationContext createInitializationContext(final File storageDirectory) { + final Map properties = new HashMap<>(); + properties.put(StandardConnectorAssetManager.ASSET_STORAGE_LOCATION_PROPERTY, storageDirectory.getAbsolutePath()); + + final AssetReferenceLookup assetReferenceLookup = mock(AssetReferenceLookup.class); + final NodeTypeProvider nodeTypeProvider = mock(NodeTypeProvider.class); + + return new StandardAssetManagerInitializationContext(assetReferenceLookup, properties, nodeTypeProvider); + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/components/connector/secrets/TestParameterProviderSecretsManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/components/connector/secrets/TestParameterProviderSecretsManager.java new file mode 100644 index 000000000000..870576010efa --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/components/connector/secrets/TestParameterProviderSecretsManager.java @@ -0,0 +1,374 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.controller.ParameterProviderNode; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterDescriptor; +import org.apache.nifi.parameter.ParameterGroup; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestParameterProviderSecretsManager { + + private static final String PROVIDER_1_ID = "provider-1-id"; + private static final String PROVIDER_1_NAME = "Provider One"; + private static final String PROVIDER_2_ID = "provider-2-id"; + private static final String PROVIDER_2_NAME = "Provider Two"; + + private static final String GROUP_1_NAME = "Group One"; + private static final String GROUP_2_NAME = "Group Two"; + + private static final String SECRET_1_NAME = "secret-one"; + private static final String SECRET_1_DESCRIPTION = "First secret"; + private static final String SECRET_1_VALUE = "secret-value-one"; + + private static final String SECRET_2_NAME = "secret-two"; + private static final String SECRET_2_DESCRIPTION = "Second secret"; + private static final String SECRET_2_VALUE = "secret-value-two"; + + private static final String SECRET_3_NAME = "secret-three"; + private static final String SECRET_3_DESCRIPTION = "Third secret"; + private static final String SECRET_3_VALUE = "secret-value-three"; + + private ParameterProviderSecretsManager secretsManager; + + @BeforeEach + public void setup() { + final FlowManager flowManager = mock(FlowManager.class); + final ParameterProviderNode providerNode1 = createMockedParameterProviderNode(PROVIDER_1_ID, PROVIDER_1_NAME, GROUP_1_NAME, + createParameter(SECRET_1_NAME, SECRET_1_DESCRIPTION, SECRET_1_VALUE), + createParameter(SECRET_2_NAME, SECRET_2_DESCRIPTION, SECRET_2_VALUE)); + final ParameterProviderNode providerNode2 = createMockedParameterProviderNode(PROVIDER_2_ID, PROVIDER_2_NAME, GROUP_2_NAME, + createParameter(SECRET_3_NAME, SECRET_3_DESCRIPTION, SECRET_3_VALUE)); + + final Set providers = new HashSet<>(); + providers.add(providerNode1); + providers.add(providerNode2); + when(flowManager.getAllParameterProviders()).thenReturn(providers); + + secretsManager = new ParameterProviderSecretsManager(); + final SecretsManagerInitializationContext initContext = new StandardSecretsManagerInitializationContext(flowManager); + secretsManager.initialize(initContext); + } + + private ParameterProviderNode createMockedParameterProviderNode(final String id, final String name, final String groupName, final Parameter... parameters) { + return createMockedParameterProviderNode(id, name, groupName, ValidationStatus.VALID, parameters); + } + + private ParameterProviderNode createMockedParameterProviderNode(final String id, final String name, final String groupName, + final ValidationStatus validationStatus, final Parameter... parameters) { + final ParameterProviderNode node = mock(ParameterProviderNode.class); + when(node.getIdentifier()).thenReturn(id); + when(node.getName()).thenReturn(name); + when(node.getValidationStatus()).thenReturn(validationStatus); + + final List parameterList = List.of(parameters); + final ParameterGroup group = new ParameterGroup(groupName, parameterList); + final List groups = List.of(group); + + when(node.fetchParameterValues()).thenReturn(groups); + + when(node.fetchParameterValues(anyList())).thenAnswer(invocation -> { + final List requestedNames = invocation.getArgument(0); + final List matchingParameters = parameterList.stream() + .filter(p -> requestedNames.contains(name + "." + groupName + "." + p.getDescriptor().getName())) + .toList(); + if (matchingParameters.isEmpty()) { + return List.of(); + } + return List.of(new ParameterGroup(groupName, matchingParameters)); + }); + + return node; + } + + private Parameter createParameter(final String name, final String description, final String value) { + final ParameterDescriptor descriptor = new ParameterDescriptor.Builder() + .name(name) + .description(description) + .build(); + return new Parameter.Builder() + .descriptor(descriptor) + .value(value) + .build(); + } + + private SecretReference createSecretReference(final String providerId, final String providerName, final String secretName) { + final String effectiveProviderId = providerId != null ? providerId : getProviderIdFromName(providerName); + final String effectiveProviderName = getProviderNameFromId(effectiveProviderId); + final String groupName = getGroupNameForProvider(effectiveProviderId); + return new SecretReference(providerId, providerName, secretName, effectiveProviderName + "." + groupName + "." + secretName); + } + + private String getProviderIdFromName(final String providerName) { + if (PROVIDER_1_NAME.equals(providerName)) { + return PROVIDER_1_ID; + } else if (PROVIDER_2_NAME.equals(providerName)) { + return PROVIDER_2_ID; + } + return null; + } + + private String getProviderNameFromId(final String providerId) { + if (PROVIDER_1_ID.equals(providerId)) { + return PROVIDER_1_NAME; + } else if (PROVIDER_2_ID.equals(providerId)) { + return PROVIDER_2_NAME; + } + return "Unknown Provider"; + } + + private String getGroupNameForProvider(final String providerId) { + if (PROVIDER_1_ID.equals(providerId)) { + return GROUP_1_NAME; + } + return GROUP_2_NAME; + } + + @Test + public void testGetSecretProvidersReturnsOneProviderPerMockedParameterProvider() { + final Set providers = secretsManager.getSecretProviders(); + + assertEquals(2, providers.size()); + + boolean foundProvider1 = false; + boolean foundProvider2 = false; + for (final SecretProvider provider : providers) { + if (PROVIDER_1_ID.equals(provider.getProviderId())) { + assertEquals(PROVIDER_1_NAME, provider.getProviderName()); + foundProvider1 = true; + } else if (PROVIDER_2_ID.equals(provider.getProviderId())) { + assertEquals(PROVIDER_2_NAME, provider.getProviderName()); + foundProvider2 = true; + } + } + + assertTrue(foundProvider1); + assertTrue(foundProvider2); + } + + @Test + public void testGetAllSecretsRetrievesSecretsFromAllProviders() { + final List allSecrets = secretsManager.getAllSecrets(); + + assertEquals(3, allSecrets.size()); + + boolean foundSecret1 = false; + boolean foundSecret2 = false; + boolean foundSecret3 = false; + + for (final Secret secret : allSecrets) { + if (SECRET_1_NAME.equals(secret.getName())) { + assertEquals(PROVIDER_1_NAME, secret.getProviderName()); + assertEquals(GROUP_1_NAME, secret.getGroupName()); + assertEquals(SECRET_1_VALUE, secret.getValue()); + foundSecret1 = true; + } else if (SECRET_2_NAME.equals(secret.getName())) { + assertEquals(PROVIDER_1_NAME, secret.getProviderName()); + assertEquals(GROUP_1_NAME, secret.getGroupName()); + assertEquals(SECRET_2_VALUE, secret.getValue()); + foundSecret2 = true; + } else if (SECRET_3_NAME.equals(secret.getName())) { + assertEquals(PROVIDER_2_NAME, secret.getProviderName()); + assertEquals(GROUP_2_NAME, secret.getGroupName()); + assertEquals(SECRET_3_VALUE, secret.getValue()); + foundSecret3 = true; + } + } + + assertTrue(foundSecret1); + assertTrue(foundSecret2); + assertTrue(foundSecret3); + } + + @Test + public void testGetSecretReturnsPopulatedOptionalWhenSecretIsFoundById() { + final SecretReference reference = createSecretReference(PROVIDER_1_ID, null, SECRET_1_NAME); + + final Optional result = secretsManager.getSecret(reference); + + assertTrue(result.isPresent()); + final Secret secret = result.get(); + assertEquals(SECRET_1_NAME, secret.getName()); + assertEquals(SECRET_1_VALUE, secret.getValue()); + assertEquals(PROVIDER_1_NAME, secret.getProviderName()); + } + + @Test + public void testGetSecretReturnsPopulatedOptionalWhenSecretIsFoundByName() { + final SecretReference reference = createSecretReference(null, PROVIDER_2_NAME, SECRET_3_NAME); + + final Optional result = secretsManager.getSecret(reference); + + assertTrue(result.isPresent()); + final Secret secret = result.get(); + assertEquals(SECRET_3_NAME, secret.getName()); + assertEquals(SECRET_3_VALUE, secret.getValue()); + assertEquals(PROVIDER_2_NAME, secret.getProviderName()); + } + + @Test + public void testGetSecretReturnsEmptyOptionalWhenInvalidSecretNameProvided() { + final SecretReference reference = createSecretReference(PROVIDER_1_ID, PROVIDER_1_NAME, "non-existent-secret"); + final Optional result = secretsManager.getSecret(reference); + assertFalse(result.isPresent()); + } + + @Test + public void testGetSecretReturnsEmptyOptionalWhenProviderNotFound() { + final SecretReference reference = createSecretReference("invalid-provider-id", "Invalid Provider", SECRET_1_NAME); + final Optional result = secretsManager.getSecret(reference); + assertFalse(result.isPresent()); + } + + @Test + public void testGetSecretsReturnsAllSecretsProperlyMappedWhenAllAreFound() { + final SecretReference reference1 = createSecretReference(PROVIDER_1_ID, PROVIDER_1_NAME, SECRET_1_NAME); + final SecretReference reference2 = createSecretReference(PROVIDER_1_ID, PROVIDER_1_NAME, SECRET_2_NAME); + final SecretReference reference3 = createSecretReference(PROVIDER_2_ID, PROVIDER_2_NAME, SECRET_3_NAME); + final Set references = Set.of(reference1, reference2, reference3); + + final Map results = secretsManager.getSecrets(references); + assertEquals(3, results.size()); + + final Secret secret1 = results.get(reference1); + assertNotNull(secret1); + assertEquals(SECRET_1_NAME, secret1.getName()); + assertEquals(SECRET_1_VALUE, secret1.getValue()); + + final Secret secret2 = results.get(reference2); + assertNotNull(secret2); + assertEquals(SECRET_2_NAME, secret2.getName()); + assertEquals(SECRET_2_VALUE, secret2.getValue()); + + final Secret secret3 = results.get(reference3); + assertNotNull(secret3); + assertEquals(SECRET_3_NAME, secret3.getName()); + assertEquals(SECRET_3_VALUE, secret3.getValue()); + } + + @Test + public void testGetSecretsReturnsNullValueForSecretWithInvalidName() { + final SecretReference validReference = createSecretReference(PROVIDER_1_ID, PROVIDER_1_NAME, SECRET_1_NAME); + final SecretReference invalidReference = createSecretReference(PROVIDER_1_ID, PROVIDER_1_NAME, "non-existent-secret"); + final Set references = Set.of(validReference, invalidReference); + + final Map results = secretsManager.getSecrets(references); + assertEquals(2, results.size()); + + final Secret validSecret = results.get(validReference); + assertNotNull(validSecret); + assertEquals(SECRET_1_NAME, validSecret.getName()); + + assertTrue(results.containsKey(invalidReference)); + assertNull(results.get(invalidReference)); + } + + @Test + public void testGetSecretsReturnsNullValueForSecretWithInvalidProvider() { + final SecretReference validReference = createSecretReference(PROVIDER_1_ID, PROVIDER_1_NAME, SECRET_1_NAME); + final SecretReference invalidProviderReference = createSecretReference("invalid-id", "Invalid Provider", SECRET_1_NAME); + final Set references = Set.of(validReference, invalidProviderReference); + + final Map results = secretsManager.getSecrets(references); + assertEquals(2, results.size()); + + final Secret validSecret = results.get(validReference); + assertNotNull(validSecret); + + assertTrue(results.containsKey(invalidProviderReference)); + assertNull(results.get(invalidProviderReference)); + } + + @Test + public void testGetSecretsFromMultipleProvidersWithMixedResults() { + final SecretReference provider1Secret1 = createSecretReference(PROVIDER_1_ID, null, SECRET_1_NAME); + final SecretReference provider1Invalid = createSecretReference(PROVIDER_1_ID, null, "invalid-secret"); + final SecretReference provider2Secret3 = createSecretReference(PROVIDER_2_ID, null, SECRET_3_NAME); + final SecretReference invalidProvider = createSecretReference("invalid-id", null, SECRET_1_NAME); + + final Set references = Set.of(provider1Secret1, provider1Invalid, provider2Secret3, invalidProvider); + final Map results = secretsManager.getSecrets(references); + assertEquals(4, results.size()); + + assertNotNull(results.get(provider1Secret1)); + assertEquals(SECRET_1_NAME, results.get(provider1Secret1).getName()); + + assertTrue(results.containsKey(provider1Invalid)); + assertNull(results.get(provider1Invalid)); + + assertNotNull(results.get(provider2Secret3)); + assertEquals(SECRET_3_NAME, results.get(provider2Secret3).getName()); + + assertTrue(results.containsKey(invalidProvider)); + assertNull(results.get(invalidProvider)); + } + + @Test + public void testGetSecretProviderSearchesByIdFirst() { + final SecretReference referenceWithBothIdAndName = createSecretReference(PROVIDER_1_ID, PROVIDER_2_NAME, SECRET_1_NAME); + final Optional result = secretsManager.getSecret(referenceWithBothIdAndName); + assertTrue(result.isPresent()); + assertEquals(PROVIDER_1_NAME, result.get().getProviderName()); + } + + @Test + public void testGetSecretProvidersFiltersOutInvalidProviders() { + final FlowManager flowManager = mock(FlowManager.class); + final ParameterProviderNode validProvider = createMockedParameterProviderNode("valid-id", "Valid Provider", "Group", + ValidationStatus.VALID, createParameter("secret", "description", "value")); + final ParameterProviderNode invalidProvider = createMockedParameterProviderNode("invalid-id", "Invalid Provider", "Group", + ValidationStatus.INVALID, createParameter("secret2", "description2", "value2")); + final ParameterProviderNode validatingProvider = createMockedParameterProviderNode("validating-id", "Validating Provider", "Group", + ValidationStatus.VALIDATING, createParameter("secret3", "description3", "value3")); + + final Set allProviders = new HashSet<>(); + allProviders.add(validProvider); + allProviders.add(invalidProvider); + allProviders.add(validatingProvider); + when(flowManager.getAllParameterProviders()).thenReturn(allProviders); + + final ParameterProviderSecretsManager manager = new ParameterProviderSecretsManager(); + manager.initialize(new StandardSecretsManagerInitializationContext(flowManager)); + + final Set secretProviders = manager.getSecretProviders(); + + assertEquals(1, secretProviders.size()); + assertEquals("valid-id", secretProviders.iterator().next().getProviderId()); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionTest.java index 6c2171c3c861..c82ee1ca5771 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionTest.java @@ -17,6 +17,7 @@ package org.apache.nifi.controller.repository; import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.controller.lifecycle.TaskTermination; import org.apache.nifi.controller.metrics.GaugeRecord; import org.apache.nifi.controller.repository.claim.ContentClaim; @@ -34,6 +35,8 @@ import org.mockito.Captor; import org.mockito.Mock; import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; @@ -48,10 +51,12 @@ import static org.mockito.ArgumentMatchers.eq; import static org.mockito.ArgumentMatchers.isA; import static org.mockito.ArgumentMatchers.isNull; +import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; @ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) class StandardProcessSessionTest { private static final String BACKOFF_PERIOD = "5 s"; @@ -116,6 +121,8 @@ void setSession() { when(repositoryContext.createContentClaimWriteCache(isA(PerformanceTracker.class))).thenReturn(contentClaimWriteCache); when(repositoryContext.getConnectable()).thenReturn(connectable); when(connectable.getIdentifier()).thenReturn(Connectable.class.getSimpleName()); + final FlowFileActivity flowFileActivity = mock(FlowFileActivity.class); + when(connectable.getFlowFileActivity()).thenReturn(flowFileActivity); session = new StandardProcessSession(repositoryContext, taskTermination, performanceTracker); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/service/StandardControllerServiceResolverTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/service/StandardControllerServiceResolverTest.java index e89075c5722e..658b50c87dc6 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/service/StandardControllerServiceResolverTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/controller/service/StandardControllerServiceResolverTest.java @@ -29,7 +29,7 @@ import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.registry.flow.FlowSnapshotContainer; import org.apache.nifi.registry.flow.RegisteredFlowSnapshot; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; @@ -57,7 +57,7 @@ public class StandardControllerServiceResolverTest { private static final String CHILD_REFERENCES_SERVICES_FROM_PARENT_LOCATION = BASE_SNAPSHOT_LOCATION + "/versioned-child-services-from-parent"; private static final String STANDARD_EXTERNAL_SERVICE_REFERENCE = BASE_SNAPSHOT_LOCATION + "/standard-external-service-reference"; - private NiFiRegistryFlowMapper flowMapper; + private VersionedComponentFlowMapper flowMapper; private ControllerServiceProvider controllerServiceProvider; private ControllerServiceApiLookup controllerServiceApiLookup; @@ -72,7 +72,7 @@ public class StandardControllerServiceResolverTest { public void setup() { Authorizer authorizer = mock(Authorizer.class); FlowManager flowManager = mock(FlowManager.class); - flowMapper = mock(NiFiRegistryFlowMapper.class); + flowMapper = mock(VersionedComponentFlowMapper.class); controllerServiceProvider = mock(ControllerServiceProvider.class); controllerServiceApiLookup = mock(ControllerServiceApiLookup.class); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/groups/StandardProcessGroupTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/groups/StandardProcessGroupTest.java index b3f888a3397f..0887de6c90d7 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/groups/StandardProcessGroupTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/groups/StandardProcessGroupTest.java @@ -130,7 +130,8 @@ void setProcessGroup() throws IOException { nodeTypeProvider, properties, statelessGroupNodeFactory, - assetManager + assetManager, + null ); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNodeTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNodeTest.java index a135ab8d66cd..e69a27a05f6c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNodeTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNodeTest.java @@ -23,7 +23,7 @@ import org.apache.nifi.flowanalysis.EnforcementPolicy; import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.validation.RuleViolation; import org.apache.nifi.validation.RuleViolationsManager; import org.junit.jupiter.api.Assertions; @@ -64,7 +64,7 @@ class FlowAnalyzingRegistryClientNodeTest { FlowManager flowManager; @Mock - NiFiRegistryFlowMapper flowMapper; + VersionedComponentFlowMapper flowMapper; @Mock InstantiatedVersionedProcessGroup nonVersionedProcessGroup; @@ -90,7 +90,7 @@ class FlowAnalyzingRegistryClientNodeTest { @BeforeEach public void setUp() { Mockito.when(versionedProcessGroup.getInstanceIdentifier()).thenReturn(INSTANCE_IDENTIFIER); - Mockito.when(flowManager.getGroup(Mockito.anyString())).thenReturn(processGroup); + Mockito.when(flowManager.getGroup(Mockito.anyString(), Mockito.eq(null))).thenReturn(processGroup); Mockito.when(flowMapper.mapNonVersionedProcessGroup(Mockito.same(processGroup), Mockito.same(serviceProvider))).thenReturn(nonVersionedProcessGroup); Mockito.when(ruleViolation1.getEnforcementPolicy()).thenReturn(EnforcementPolicy.ENFORCE); Mockito.when(ruleViolation2.getEnforcementPolicy()).thenReturn(EnforcementPolicy.ENFORCE); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/mapping/TestNiFiRegistryFlowMapper.java b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/mapping/TestVersionedComponentFlowMapper.java similarity index 95% rename from nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/mapping/TestNiFiRegistryFlowMapper.java rename to nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/mapping/TestVersionedComponentFlowMapper.java index cd08afd42304..cf2635927781 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/mapping/TestNiFiRegistryFlowMapper.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/test/java/org/apache/nifi/registry/flow/mapping/TestVersionedComponentFlowMapper.java @@ -55,14 +55,14 @@ import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; -public class TestNiFiRegistryFlowMapper { +public class TestVersionedComponentFlowMapper { private static final SensitiveValueEncryptor ENCRYPTOR = value -> new StringBuilder(value).reverse().toString(); @Test public void testMappingProcessorWithSensitiveValuesGivesNullValue() { final ExtensionManager extensionManager = mock(ExtensionManager.class); final FlowMappingOptions mappingOptions = FlowMappingOptions.DEFAULT_OPTIONS; - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(extensionManager, mappingOptions); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(extensionManager, mappingOptions); final ControllerServiceProvider serviceProvider = mock(ControllerServiceProvider.class); final Map properties = new HashMap<>(); @@ -82,7 +82,7 @@ public void testMappingProcessorWithSensitiveValuesGivesNullValue() { public void testMappingProcessorWithSensitiveValuesLeavesSensitiveParameterReference() { final ExtensionManager extensionManager = mock(ExtensionManager.class); final FlowMappingOptions mappingOptions = FlowMappingOptions.DEFAULT_OPTIONS; - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(extensionManager, mappingOptions); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(extensionManager, mappingOptions); final ControllerServiceProvider serviceProvider = mock(ControllerServiceProvider.class); final Map properties = new HashMap<>(); @@ -108,7 +108,7 @@ public void testMappingProcessorWithSensitiveValuesProvidesEncryptedValue() { .sensitiveValueEncryptor(ENCRYPTOR) .build(); - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(extensionManager, mappingOptions); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(extensionManager, mappingOptions); final ControllerServiceProvider serviceProvider = mock(ControllerServiceProvider.class); final Map properties = new HashMap<>(); @@ -142,7 +142,7 @@ private void testControllerServicesMapsToProperId(final boolean useVersionedId) .mapControllerServiceReferencesToVersionedId(useVersionedId) .build(); - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(extensionManager, mappingOptions); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(extensionManager, mappingOptions); final ControllerServiceNode mockServiceNode = mock(ControllerServiceNode.class); when(mockServiceNode.getIdentifier()).thenReturn("1234"); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConfigurationUpdateResult.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConfigurationUpdateResult.java new file mode 100644 index 000000000000..0a9922219ee8 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConfigurationUpdateResult.java @@ -0,0 +1,24 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +public enum ConfigurationUpdateResult { + NO_CHANGES, + + CHANGES_MADE; +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorAction.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorAction.java new file mode 100644 index 000000000000..60a4865269a4 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorAction.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +/** + * Represents an action that can be performed on a Connector. + */ +public interface ConnectorAction { + + /** + * Returns the name of this action. + * @return the action name + */ + String getName(); + + /** + * Returns a description of what this action does. + * @return the action description + */ + String getDescription(); + + /** + * Returns whether this action is currently allowed to be performed. + * @return true if the action is allowed, false otherwise + */ + boolean isAllowed(); + + /** + * Returns the reason why this action is not allowed, or null if the action is allowed. + * @return the reason the action is not allowed, or null if allowed + */ + String getReasonNotAllowed(); +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfiguration.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfiguration.java new file mode 100644 index 000000000000..4261e2af3d20 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorConfiguration.java @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.components.connector; + +import java.util.HashSet; +import java.util.Objects; +import java.util.Set; + +public class ConnectorConfiguration { + + private final Set stepConfigurations; + + public ConnectorConfiguration(final Set stepConfigurations) { + this.stepConfigurations = new HashSet<>(stepConfigurations); + } + + public Set getNamedStepConfigurations() { + return stepConfigurations; + } + + public NamedStepConfiguration getNamedStepConfiguration(final String name) { + for (final NamedStepConfiguration stepConfiguration : stepConfigurations) { + if (stepConfiguration.stepName().equals(name)) { + return stepConfiguration; + } + } + return null; + } + + @Override + public boolean equals(final Object other) { + if (other == null || getClass() != other.getClass()) { + return false; + } + + final ConnectorConfiguration that = (ConnectorConfiguration) other; + return Objects.equals(stepConfigurations, that.stepConfigurations); + } + + @Override + public int hashCode() { + return Objects.hashCode(stepConfigurations); + } + + @Override + public String toString() { + return "ConnectorConfiguration[" + + "stepConfigurations=" + stepConfigurations + + "]"; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorNode.java new file mode 100644 index 000000000000..bec52c287547 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorNode.java @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.authorization.resource.ComponentAuthorizable; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.VersionedComponent; +import org.apache.nifi.components.validation.ValidationState; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.components.validation.ValidationTrigger; +import org.apache.nifi.connectable.FlowFileTransferCounts; +import org.apache.nifi.engine.FlowEngine; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConfigurationStep; +import org.apache.nifi.logging.ComponentLog; + +import java.time.Duration; +import java.util.Collection; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.Future; + +public interface ConnectorNode extends ComponentAuthorizable, VersionedComponent { + + String getName(); + + ConnectorState getCurrentState(); + + ConnectorState getDesiredState(); + + /** + * Verifies that the Connector is in a state that allows it to be deleted. If the Connector is not in a state + * that allows it to be deleted, this method will throw an IllegalStateException. + * + * @throws IllegalStateException if the Connector is not in a state that allows it to be deleted + */ + void verifyCanDelete(); + + void verifyCanStart(); + + Connector getConnector(); + + /** + * @return the fully qualified class name of the underlying Connector implementation + */ + String getComponentType(); + + String getCanonicalClassName(); + + BundleCoordinate getBundleCoordinate(); + + /** + * Returns whether or not the underlying extension is missing (i.e., the Connector is a GhostConnector). + * @return true if the extension is missing, false otherwise + */ + boolean isExtensionMissing(); + + List fetchAllowableValues(String stepName, String propertyName); + + List fetchAllowableValues(String stepName, String propertyName, String filter); + + void initializeConnector(FrameworkConnectorInitializationContext initializationContext); + + void loadInitialFlow() throws FlowUpdateException; + + /** + *

+ * Pause triggering asynchronous validation to occur when the connector is updated. Often times, it is necessary + * to update several aspects of a connector, such as the properties and annotation data, at once. When this occurs, + * we don't want to trigger validation for each update, so we can follow the pattern: + *

+ * + * 
+     * 
+     * connectorNode.pauseValidationTrigger();
+     * try {
+     *   connectorNode.setProperties(properties);
+     *   connectorNode.setAnnotationData(annotationData);
+     * } finally {
+     *   connectorNode.resumeValidationTrigger();
+     * }
+     * 
+     *
+ * + *

+ * When calling this method, it is imperative that {@link #resumeValidationTrigger()} is always called within a {@code finally} block to + * ensure that validation occurs. + *

+ */ + void pauseValidationTrigger(); + + /** + * Resume triggering asynchronous validation to occur when the connector is updated. This method is to be used in conjunction + * with {@link #pauseValidationTrigger()} as illustrated in its documentation. When this method is called, if the connector's Validation Status + * is {@link ValidationStatus#VALIDATING}, connector validation will immediately be triggered asynchronously. + */ + void resumeValidationTrigger(); + + /** + * Indicates whether validation triggering is currently paused. + * @return true if validation triggering is paused, false otherwise + */ + boolean isValidationPaused(); + + List verifyConfigurationStep(String configurationStepName, StepConfiguration configurationOverrides); + + List verify(); + + ComponentLog getComponentLog(); + + List getConfigurationSteps(); + + FrameworkFlowContext getActiveFlowContext(); + + FrameworkFlowContext getWorkingFlowContext(); + + void discardWorkingConfiguration(); + + // ------------------- + // The following methods should always be called via the ConnectorRepository in order to maintain proper + // lifecycle management of the Connector. + + /** + * Sets the name of the Connector. This method should only be invoked via the ConnectorRepository. + * @param name the Connector's name + */ + void setName(String name); + + /** + * Performs validation logic that is defined by the Connector. + * @return the ValidationState indicating the results of the validation + */ + ValidationState performValidation(); + + /** + * Triggers validation of all processors and controller services within the Connector's managed ProcessGroup. + * + * @param validationTrigger the ValidationTrigger to use for triggering component validation + */ + void validateComponents(ValidationTrigger validationTrigger); + + /** + * Returns the current validation status of the connector. + * @return the current ValidationStatus + */ + ValidationStatus getValidationStatus(); + + /** + * Returns the validation errors for the connector, or an empty collection if the connector is valid. + * @return the validation errors, or an empty collection if valid + */ + Collection getValidationErrors(); + + + ValidationState getValidationState(); + + /** + * Returns an Optional Duration indicating how long the Connector has been idle (i.e., not processed any FlowFiles and with no FlowFiles queued). + * If there is any FlowFile queued, the Optional will be empty. Otherwise, the duration will indicate how long it has been since the + * last FlowFile was acted upon. + * + * @return an Optional Duration indicating how long the Connector has been idle + */ + Optional getIdleDuration(); + + /** + * Returns the FlowFileTransferCounts that represents that amount of data sent and received + * by this Connector since it was started. + * @return the FlowFileTransferCounts for this Connector + */ + FlowFileTransferCounts getFlowFileTransferCounts(); + + /** + * Starts the Connector. This method should only be invoked via the ConnectorRepository. + * @param scheduler the ScheduledExecutorService to use for scheduling any tasks that the Connector needs to perform + * @return a Future that will be completed when the Connector has started + */ + Future start(FlowEngine scheduler); + + /** + * Stops the Connector. This method should only be invoked via the ConnectorRepository. + * @param scheduler the ScheduledExecutorService to use for scheduling any tasks that the Connector needs to perform + * @return a Future that will be completed when the Connector has stopped + */ + Future stop(FlowEngine scheduler); + + /** + * Allows the Connector to drain any in-flight data while not accepting any new data. + */ + Future drainFlowFiles(); + + /** + * Cancels the draining of FlowFiles that is currently in progress. + * @throws IllegalStateException if the Connector is not currently draining FlowFiles + */ + void cancelDrainFlowFiles(); + + /** + * Verifies that the Connector can cancel draining FlowFiles. + * @throws IllegalStateException if not in a state where draining FlowFiles can be cancelled + */ + void verifyCancelDrainFlowFiles() throws IllegalStateException; + + /** + * Verifies that the Connector can have its FlowFiles purged. + * @throws IllegalStateException if not in a state where FlowFiles can be purged + */ + void verifyCanPurgeFlowFiles() throws IllegalStateException; + + /** + * Purges all FlowFiles from the Connector, immediately dropping the data. + * + * @param requestor the user requesting the purge. This will be recorded in the associated provenance DROP events. + * @return a Future that will be completed when the purge operation is finished + */ + Future purgeFlowFiles(String requestor); + + /** + * Updates the configuration of one of the configuration steps. This method should only be invoked via the ConnectorRepository. + * @param configurationStepName the name of the configuration step being set + * (must match one of the names returned by {@link Connector#getConfigurationSteps()}) + * @param configuration the configuration for the given configuration step + * @throws FlowUpdateException if unable to apply the configuration changes + */ + void setConfiguration(String configurationStepName, StepConfiguration configuration) throws FlowUpdateException; + + void transitionStateForUpdating(); + + void prepareForUpdate() throws FlowUpdateException; + + /** + * Aborts the update preparation process. This method should only be invoked via the ConnectorRepository. + * @param cause the reason for aborting the update preparation + */ + // TODO: Should this return a Future? + void abortUpdate(Throwable cause); + + /** + * Notifies the Connector that the update process is finished and it can apply any changes that were made during the + * update process. This method should only be invoked via the ConnectorRepository. + * @throws FlowUpdateException if unable to apply the changes made during the update process + */ + void applyUpdate() throws FlowUpdateException; + + /** + * Inherits the given flow configuration into this Connector's active flow configuration. + * @param activeFlowConfiguration the active flow configuration to inherit + * @param workingFlowConfiguration the working flow configuration to inherit + * @param flowContextBundle the bundle associated with the provided configuration + * @throws FlowUpdateException if unable to inherit the given flow configuration + * @throws IllegalStateException if the Connector is not in a state of UPDATING + */ + void inheritConfiguration(List activeFlowConfiguration, List workingFlowConfiguration, + Bundle flowContextBundle) throws FlowUpdateException; + + /** + * Marks the connector as invalid with the given subject and explanation. This is used when a flow update + * fails during initialization or flow synchronization to indicate the connector cannot operate. + * + * @param subject the subject of the validation failure + * @param explanation the reason the connector is invalid + */ + void markInvalid(final String subject, final String explanation); + + /** + * Returns the list of available actions that can be performed on this Connector. + * Each action includes whether it is currently allowed and, if not, the reason why. + * @return the list of available actions + */ + List getAvailableActions(); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRepository.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRepository.java new file mode 100644 index 000000000000..cfa5ae5f9a68 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRepository.java @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConfigurationStep; + +import java.io.IOException; +import java.io.InputStream; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.Future; + +public interface ConnectorRepository { + + void initialize(ConnectorRepositoryInitializationContext context); + + /** + * Verifies that a connector with the given identifier can be created. This checks that the connector + * does not already exist in the repository, and if a ConnectorConfigurationProvider is configured, + * delegates to the provider's verifyCreate method to ensure the external store can support the operation. + * + * @param connectorId the identifier of the connector to be created + * @throws IllegalStateException if a connector with the given identifier already exists + * @throws ConnectorConfigurationProviderException if the provider rejects the create operation + */ + void verifyCreate(String connectorId); + + /** + * Adds the given Connector to the Repository + * @param connector the Connector to add + */ + void addConnector(ConnectorNode connector); + + /** + * Restores a previously added Connector to the Repository on restart. + * This is differentiated from addConnector in that this method is not called + * for newly created Connectors during the typical lifecycle of NiFi, but rather + * only to notify the Repository of Connectors that were present when NiFi was last shutdown. + * + * @param connector the Connector to restore + */ + void restoreConnector(ConnectorNode connector); + + /** + * Removes the given Connector from the Repository + * @param connectorId the identifier of the Connector to remove + */ + void removeConnector(String connectorId); + + /** + * Gets the Connector with the given identifier + * @param identifier the identifier of the Connector to get + * @return the Connector with the given identifier, or null if no such Connector exists + */ + ConnectorNode getConnector(String identifier); + + /** + * @return all Connectors in the Repository + */ + List getConnectors(); + + /** + * Starts the given Connector, managing any appropriate lifecycle events. + * @param connector the Connector to start + * @return a CompletableFuture that will be completed when the Connector has started + */ + Future startConnector(ConnectorNode connector); + + /** + * Stops the given Connector, managing any appropriate lifecycle events. + * @param connector the Connector to stop + * @return a CompletableFuture that will be completed when the Connector has stopped + */ + Future stopConnector(ConnectorNode connector); + + /** + * Restarts the given Connector, managing any appropriate lifecycle events. + * + * @param connector the Connector to restart + * @return a CompletableFuture that will be completed when the Connector has restarted + */ + Future restartConnector(ConnectorNode connector); + + /** + * Updates the metadata of a Connector, such as its name. This method should be used instead of calling + * {@link ConnectorNode#setName(String)} directly, so that the ConnectorRepository can synchronize + * changes with the ConnectorConfigurationProvider if one is configured. + * + * @param connector the Connector to update + * @param name the new name for the Connector + */ + void updateConnector(ConnectorNode connector, String name); + + void configureConnector(ConnectorNode connector, String stepName, StepConfiguration configuration) throws FlowUpdateException; + + void applyUpdate(ConnectorNode connector, ConnectorUpdateContext context) throws FlowUpdateException; + + void inheritConfiguration(ConnectorNode connector, List activeFlowConfiguration, + List workingFlowConfiguration, Bundle flowContextBundle) throws FlowUpdateException; + + void discardWorkingConfiguration(ConnectorNode connector); + + SecretsManager getSecretsManager(); + + /** + * Creates a new ConnectorStateTransition instance for managing the lifecycle state of a connector. + * + * @param type the connector type + * @param id the connector identifier + * @return a new ConnectorStateTransition instance + */ + ConnectorStateTransition createStateTransition(String type, String id); + + FrameworkConnectorInitializationContextBuilder createInitializationContextBuilder(); + + /** + * Stores an asset for the given connector. If a {@link ConnectorConfigurationProvider} is configured, + * the asset is also uploaded to the external store and a mapping between the NiFi asset ID and the + * external ID is recorded. If the provider upload fails, the local asset is rolled back. + * + * @param connectorId the identifier of the connector that owns the asset + * @param assetId the NiFi-assigned asset identifier (UUID) + * @param assetName the filename of the asset + * @param content the binary content of the asset + * @return the stored Asset + * @throws IOException if an I/O error occurs during storage + */ + Asset storeAsset(String connectorId, String assetId, String assetName, InputStream content) throws IOException; + + /** + * Retrieves an asset by its NiFi-assigned identifier. + * + * @param assetId the NiFi-assigned asset identifier (UUID) + * @return an Optional containing the asset if found, or empty if no asset exists with the given ID + */ + Optional getAsset(String assetId); + + /** + * Retrieves all assets belonging to the given connector. + * + * @param connectorId the identifier of the connector + * @return the list of assets for the connector + */ + List getAssets(String connectorId); + + /** + * Deletes all assets belonging to the given connector from the local store and, + * if a {@link ConnectorConfigurationProvider} is configured, from the external store as well. + * + * @param connectorId the identifier of the connector whose assets should be deleted + */ + void deleteAssets(String connectorId); + + /** + * Ensures that asset binaries for the given connector are available locally by downloading + * any missing or changed assets from the external {@link ConnectorConfigurationProvider}. + * This is a no-op if no provider is configured. This method should be called before operations + * that need local asset files to be present, such as configuration verification. + * + * @param connector the connector whose assets should be synced + */ + void syncAssetsFromProvider(ConnectorNode connector); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRepositoryInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRepositoryInitializationContext.java new file mode 100644 index 000000000000..91bca1fcc95c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRepositoryInitializationContext.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.controller.NodeTypeProvider; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.nar.ExtensionManager; + +public interface ConnectorRepositoryInitializationContext { + + FlowManager getFlowManager(); + + ExtensionManager getExtensionManager(); + + SecretsManager getSecretsManager(); + + AssetManager getAssetManager(); + + NodeTypeProvider getNodeTypeProvider(); + + ConnectorRequestReplicator getRequestReplicator(); + + /** + * Returns the ConnectorConfigurationProvider to use for external management of connector working configuration, + * or null if no provider is configured. + * + * @return the ConnectorConfigurationProvider, or null if not configured + */ + default ConnectorConfigurationProvider getConnectorConfigurationProvider() { + return null; + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestReplicator.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestReplicator.java new file mode 100644 index 000000000000..dd6d7f90c2e0 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorRequestReplicator.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.controller.flow.FlowManager; + +import java.io.IOException; + +public interface ConnectorRequestReplicator { + + ConnectorState getState(String connectorId) throws IOException; + + void setFlowManager(FlowManager flowManager); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorState.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorState.java new file mode 100644 index 000000000000..c13999c4fa05 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorState.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +public enum ConnectorState { + STARTING, + RUNNING, + STOPPING, + STOPPED, + DRAINING, + PURGING, + PREPARING_FOR_UPDATE, + UPDATING, + UPDATE_FAILED, + UPDATED; +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorStateTransition.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorStateTransition.java new file mode 100644 index 000000000000..a9693f8eafc3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorStateTransition.java @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.util.concurrent.CompletableFuture; + +/** + * Manages the state transitions and lifecycle of a ConnectorNode, including tracking current and desired states, + * handling asynchronous state transition completion, and coordinating update resume states. + */ +public interface ConnectorStateTransition { + + /** + * Returns the current state of the connector. + * + * @return the current ConnectorState + */ + ConnectorState getCurrentState(); + + /** + * Returns the desired state of the connector. + * + * @return the desired ConnectorState + */ + ConnectorState getDesiredState(); + + /** + * Sets the desired state for the connector. This operation is used to indicate what state the connector + * should be in, and the framework will work to transition to that state. + * + * @param desiredState the target state for the connector + */ + void setDesiredState(ConnectorState desiredState); + + /** + * Attempts to atomically update the current state from the expected state to the new state. + * + * @param expectedState the expected current state + * @param newState the new state to transition to + * @return true if the state was successfully updated, false if the current state did not match the expected state + */ + boolean trySetCurrentState(ConnectorState expectedState, ConnectorState newState); + + /** + * Sets the current state to the specified state, regardless of the previous state. + * + * @param newState the new current state + */ + void setCurrentState(ConnectorState newState); + + /** + * Registers a future to be completed when the connector transitions to the RUNNING state. + * This method is thread-safe and handles internal synchronization. + * + * @param future the CompletableFuture to complete when the connector starts + */ + void addPendingStartFuture(CompletableFuture future); + + /** + * Registers a future to be completed when the connector transitions to the STOPPED state. + * This method is thread-safe and handles internal synchronization. + * + * @param future the CompletableFuture to complete when the connector stops + */ + void addPendingStopFuture(CompletableFuture future); +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorUpdateContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorUpdateContext.java new file mode 100644 index 000000000000..c5a00bb61b8d --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorUpdateContext.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.io.IOException; + +public interface ConnectorUpdateContext { + + void saveFlow() throws IOException; + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorValidationTrigger.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorValidationTrigger.java new file mode 100644 index 000000000000..f1e617be3252 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorValidationTrigger.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +/** + * Trigger for initiating validation of a ConnectorNode. + */ +public interface ConnectorValidationTrigger { + + /** + * Triggers validation of the given connector to occur asynchronously. + * If the Connector's validation is already in progress by another thread, this method will + * return without triggering another validation. If the Connector's validation is paused, this + * will schedule the validation to occur once unpaused. + * + * @param connector the connector to validate + */ + void triggerAsync(ConnectorNode connector); + + /** + * Triggers validation of the given connector immediately in the current thread. This will + * trigger validation even if other validation is currently in progress or if the Connector's + * validation is paused. + * + * @param connector the connector to validate + */ + void trigger(ConnectorNode connector); +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FlowContextFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FlowContextFactory.java new file mode 100644 index 000000000000..c4a2cca21772 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FlowContextFactory.java @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.logging.ComponentLog; + +public interface FlowContextFactory { + FrameworkFlowContext createActiveFlowContext(String connectorId, ComponentLog connectorLogger, Bundle bundle); + + FrameworkFlowContext createWorkingFlowContext(String connectorId, ComponentLog connectorLogger, MutableConnectorConfigurationContext currentConfiguration, Bundle bundle); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkConnectorInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkConnectorInitializationContext.java new file mode 100644 index 000000000000..8749e48192b6 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkConnectorInitializationContext.java @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; + +public interface FrameworkConnectorInitializationContext extends ConnectorInitializationContext { + + SecretsManager getSecretsManager(); + + AssetManager getAssetManager(); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkConnectorInitializationContextBuilder.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkConnectorInitializationContextBuilder.java new file mode 100644 index 000000000000..9ef92941a8f6 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkConnectorInitializationContextBuilder.java @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.logging.ComponentLog; + +public interface FrameworkConnectorInitializationContextBuilder { + + FrameworkConnectorInitializationContextBuilder identifier(String identifier); + + FrameworkConnectorInitializationContextBuilder name(String name); + + FrameworkConnectorInitializationContextBuilder componentLog(ComponentLog componentLog); + + FrameworkConnectorInitializationContextBuilder assetManager(AssetManager assetManager); + + FrameworkConnectorInitializationContextBuilder secretsManager(SecretsManager secretsManager); + + FrameworkConnectorInitializationContextBuilder componentBundleLookup(ComponentBundleLookup bundleLookup); + + FrameworkConnectorInitializationContext build(); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkFlowContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkFlowContext.java new file mode 100644 index 000000000000..2c05207bf87b --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FrameworkFlowContext.java @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.groups.ProcessGroup; + +public interface FrameworkFlowContext extends FlowContext { + ProcessGroup getManagedProcessGroup(); + + @Override + MutableConnectorConfigurationContext getConfigurationContext(); + + void updateFlow(VersionedExternalFlow versionedExternalFlow, AssetManager assetManager) throws FlowUpdateException; +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/MutableConnectorConfigurationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/MutableConnectorConfigurationContext.java new file mode 100644 index 000000000000..22191633af01 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/MutableConnectorConfigurationContext.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.util.Map; + +public interface MutableConnectorConfigurationContext extends ConnectorConfigurationContext { + + /** + * Sets the properties for the given step to the provided properties. Any existing properties + * for the step that are not included in the provided configuration will remain unchanged. + * + * @param stepName the name of the configuration step + * @param configuration the configuration to set + * @return the result of the configuration update + */ + ConfigurationUpdateResult setProperties(String stepName, StepConfiguration configuration); + + /** + * Replaces all of the properties for the given step with the provided properties. Any existing properties + * for the step that are not included in the provided configuration will be removed. + * + * @param stepName the name of the configuration step + * @param configuration the configuration to set + * @return the result of the configuration update + */ + ConfigurationUpdateResult replaceProperties(String stepName, StepConfiguration configuration); + + /** + * Resolves all existing property values. + */ + void resolvePropertyValues(); + + /** + * Converts this mutable configuration context to an immutable ConnectorConfiguration. + * @return the ConnectorConfiguration + */ + ConnectorConfiguration toConnectorConfiguration(); + + @Override + MutableConnectorConfigurationContext createWithOverrides(String stepName, Map propertyValues); + + /** + * Creates a clone of this MutableConnectorConfigurationContext. + * @return the cloned context + */ + @Override + MutableConnectorConfigurationContext clone(); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/NamedStepConfiguration.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/NamedStepConfiguration.java new file mode 100644 index 000000000000..7c17798ac81a --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/NamedStepConfiguration.java @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +public record NamedStepConfiguration(String stepName, StepConfiguration configuration) { +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ParameterContextFacadeFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ParameterContextFacadeFactory.java new file mode 100644 index 000000000000..7ac6b66a7a93 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ParameterContextFacadeFactory.java @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.groups.ProcessGroup; + +public interface ParameterContextFacadeFactory { + ParameterContextFacade create(ProcessGroup processGroup); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ProcessGroupFacadeFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ProcessGroupFacadeFactory.java new file mode 100644 index 000000000000..e8d2189c7219 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ProcessGroupFacadeFactory.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; + +public interface ProcessGroupFacadeFactory { + ProcessGroupFacade create(ProcessGroup processGroup, ComponentLog connectorLogger); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ProcessGroupFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ProcessGroupFactory.java new file mode 100644 index 000000000000..5e02a794bbae --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ProcessGroupFactory.java @@ -0,0 +1,24 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.groups.ProcessGroup; + +public interface ProcessGroupFactory { + ProcessGroup create(String id); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretProvider.java new file mode 100644 index 000000000000..3c0a2f4e5f16 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretProvider.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.components.connector.Secret; + +import java.util.List; + +public interface SecretProvider { + + String getProviderId(); + + String getProviderName(); + + List getAllSecrets(); + + List getSecrets(List fullyQualifiedSecretNames); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretsManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretsManager.java new file mode 100644 index 000000000000..62365ed940e7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretsManager.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.SecretReference; + +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +public interface SecretsManager { + + void initialize(SecretsManagerInitializationContext initializationContext); + + List getAllSecrets(); + + Set getSecretProviders(); + + Optional getSecret(SecretReference secretReference); + + Map getSecrets(Set secretReferences); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretsManagerInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretsManagerInitializationContext.java new file mode 100644 index 000000000000..4f4cf8e9a861 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/secrets/SecretsManagerInitializationContext.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.secrets; + +import org.apache.nifi.controller.flow.FlowManager; + +public interface SecretsManagerInitializationContext { + + FlowManager getFlowManager(); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/DisabledServiceValidationResult.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/DisabledServiceValidationResult.java index 7cf7c31909a4..a1895b27b8c9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/DisabledServiceValidationResult.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/DisabledServiceValidationResult.java @@ -19,7 +19,10 @@ import org.apache.nifi.components.ValidationResult; +import java.util.regex.Pattern; + public class DisabledServiceValidationResult extends ValidationResult { + private static final Pattern MESSAGE_PATTERN = Pattern.compile("Controller Service with ID (.+) is disabled"); private String serviceId; public DisabledServiceValidationResult(final String subject, final String serviceId) { @@ -39,4 +42,15 @@ public DisabledServiceValidationResult(final String subject, final String servic public String getControllerServiceIdentifier() { return serviceId; } + + public static boolean isMatch(final ValidationResult result) { + if (result instanceof DisabledServiceValidationResult) { + return true; + } + if (result.isValid()) { + return false; + } + + return MESSAGE_PATTERN.matcher(result.getExplanation()).matches(); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/ValidationState.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/ValidationState.java index 184f891d455c..d1efd43ab9c3 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/ValidationState.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/validation/ValidationState.java @@ -20,14 +20,26 @@ import org.apache.nifi.components.ValidationResult; import java.util.Collection; +import java.util.Collections; public class ValidationState { private final ValidationStatus status; private final Collection validationErrors; - public ValidationState(final ValidationStatus status, final Collection validationErrors) { + public ValidationState(final ValidationStatus status, final Collection validationResults) { this.status = status; - this.validationErrors = validationErrors; + // Ensure that if we are provided any valid results, they are filtered out because we only want to store validation failures. + this.validationErrors = removeValidResults(validationResults); + } + + private Collection removeValidResults(final Collection validationResults) { + if (validationResults.isEmpty()) { + return Collections.emptyList(); + } + + return validationResults.stream() + .filter(vr -> !vr.isValid()) + .toList(); } public ValidationStatus getStatus() { diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/Connectable.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/Connectable.java index 669c6f3c837a..f320d9ed550e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/Connectable.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/Connectable.java @@ -314,4 +314,6 @@ default boolean isSessionBatchingSupported() { void setMaxBackoffPeriod(String maxBackoffPeriod); String evaluateParameters(String value); + + FlowFileActivity getFlowFileActivity(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/ConnectableFlowFileActivity.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/ConnectableFlowFileActivity.java new file mode 100644 index 000000000000..bb4373da5915 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/ConnectableFlowFileActivity.java @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectable; + +import java.util.OptionalLong; +import java.util.concurrent.atomic.AtomicReference; + +public class ConnectableFlowFileActivity implements FlowFileActivity { + private volatile long latestActivityTime = -1L; + private final AtomicReference transferCounts = new AtomicReference<>(new FlowFileTransferCounts()); + + @Override + public void updateLatestActivityTime() { + latestActivityTime = System.currentTimeMillis(); + } + + @Override + public void updateTransferCounts(final int receivedCount, final long receivedBytes, final int sentCount, final long sentBytes) { + if (receivedCount < 0) { + throw new IllegalArgumentException("Received count cannot be negative: " + receivedCount); + } + if (receivedBytes < 0L) { + throw new IllegalArgumentException("Received bytes cannot be negative: " + receivedBytes); + } + if (sentCount < 0) { + throw new IllegalArgumentException("Sent count cannot be negative: " + sentCount); + } + if (sentBytes < 0L) { + throw new IllegalArgumentException("Sent bytes cannot be negative: " + sentBytes); + } + + boolean updated = false; + while (!updated) { + final FlowFileTransferCounts currentCounts = transferCounts.get(); + final FlowFileTransferCounts updatedCounts = currentCounts.plus(receivedCount, receivedBytes, sentCount, sentBytes); + updated = transferCounts.compareAndSet(currentCounts, updatedCounts); + } + } + + @Override + public void reset() { + latestActivityTime = -1L; + transferCounts.set(new FlowFileTransferCounts()); + } + + @Override + public OptionalLong getLatestActivityTime() { + return latestActivityTime == -1L ? OptionalLong.empty() : OptionalLong.of(latestActivityTime); + } + + @Override + public FlowFileTransferCounts getTransferCounts() { + return transferCounts.get(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/FlowFileActivity.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/FlowFileActivity.java new file mode 100644 index 000000000000..50610c138770 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/FlowFileActivity.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectable; + +import java.util.OptionalLong; + +public interface FlowFileActivity { + + void updateLatestActivityTime(); + + void updateTransferCounts(int receivedCount, long receivedBytes, int sentCount, long sentBytes); + + void reset(); + + OptionalLong getLatestActivityTime(); + + FlowFileTransferCounts getTransferCounts(); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/FlowFileTransferCounts.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/FlowFileTransferCounts.java new file mode 100644 index 000000000000..74b604d5bc33 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/FlowFileTransferCounts.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectable; + +public class FlowFileTransferCounts { + private final long receivedCount; + private final long receivedBytes; + private final long sentCount; + private final long sentBytes; + + public FlowFileTransferCounts() { + this(0, 0L, 0, 0L); + } + + public FlowFileTransferCounts(final long receivedCount, final long receivedBytes, final long sentCount, final long sentBytes) { + this.receivedCount = receivedCount; + this.receivedBytes = receivedBytes; + this.sentCount = sentCount; + this.sentBytes = sentBytes; + } + + public long getReceivedCount() { + return receivedCount; + } + + public long getReceivedBytes() { + return receivedBytes; + } + + public long getSentCount() { + return sentCount; + } + + public long getSentBytes() { + return sentBytes; + } + + public FlowFileTransferCounts plus(final long additionalReceivedCount, final long additionalReceivedBytes, final long additionalSentCount, final long additionalSentBytes) { + return new FlowFileTransferCounts( + this.receivedCount + additionalReceivedCount, + this.receivedBytes + additionalReceivedBytes, + this.sentCount + additionalSentCount, + this.sentBytes + additionalSentBytes + ); + } + + public FlowFileTransferCounts plus(final FlowFileTransferCounts other) { + return new FlowFileTransferCounts( + this.receivedCount + other.getReceivedCount(), + this.receivedBytes + other.getReceivedBytes(), + this.sentCount + other.getSentCount(), + this.sentBytes + other.getSentBytes() + ); + } + + public FlowFileTransferCounts minus(final FlowFileTransferCounts other) { + return new FlowFileTransferCounts( + this.receivedCount - other.getReceivedCount(), + this.receivedBytes - other.getReceivedBytes(), + this.sentCount - other.getSentCount(), + this.sentBytes - other.getSentBytes() + ); + } + + public FlowFileTransferCounts minus(final long additionalReceivedCount, final long additionalReceivedBytes, final long additionalSentCount, final long additionalSentBytes) { + return new FlowFileTransferCounts( + this.receivedCount - additionalReceivedCount, + this.receivedBytes - additionalReceivedBytes, + this.sentCount - additionalSentCount, + this.sentBytes - additionalSentBytes + ); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/ProcessGroupFlowFileActivity.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/ProcessGroupFlowFileActivity.java new file mode 100644 index 000000000000..1d8449ef47a2 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/connectable/ProcessGroupFlowFileActivity.java @@ -0,0 +1,170 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectable; + +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.groups.StatelessGroupNode; + +import java.util.Optional; +import java.util.OptionalLong; +import java.util.concurrent.atomic.AtomicLong; + +public class ProcessGroupFlowFileActivity implements FlowFileActivity { + private final ProcessGroup group; + + public ProcessGroupFlowFileActivity(final ProcessGroup group) { + this.group = group; + } + + @Override + public void updateLatestActivityTime() { + throw new UnsupportedOperationException(); + } + + @Override + public void updateTransferCounts(final int receivedCount, final long receivedBytes, final int sentCount, final long sentBytes) { + throw new UnsupportedOperationException(); + } + + @Override + public void reset() { + } + + @Override + public OptionalLong getLatestActivityTime() { + final AtomicLong latestActivityTime = new AtomicLong(-1L); + + // Check all processors + for (final Connectable connectable : group.getProcessors()) { + final OptionalLong activityTime = connectable.getFlowFileActivity().getLatestActivityTime(); + activityTime.ifPresent(time -> { + if (time > latestActivityTime.get()) { + latestActivityTime.set(time); + } + }); + } + + // Check all input ports + for (final Connectable connectable : group.getInputPorts()) { + final OptionalLong activityTime = connectable.getFlowFileActivity().getLatestActivityTime(); + activityTime.ifPresent(time -> { + if (time > latestActivityTime.get()) { + latestActivityTime.set(time); + } + }); + } + + // Check all output ports + for (final Connectable connectable : group.getOutputPorts()) { + final OptionalLong activityTime = connectable.getFlowFileActivity().getLatestActivityTime(); + activityTime.ifPresent(time -> { + if (time > latestActivityTime.get()) { + latestActivityTime.set(time); + } + }); + } + + // Check all funnels + for (final Connectable connectable : group.getFunnels()) { + final OptionalLong activityTime = connectable.getFlowFileActivity().getLatestActivityTime(); + activityTime.ifPresent(time -> { + if (time > latestActivityTime.get()) { + latestActivityTime.set(time); + } + }); + } + + // Check stateless group node if present + final Optional statelessGroupNode = group.getStatelessGroupNode(); + if (statelessGroupNode.isPresent()) { + final OptionalLong activityTime = statelessGroupNode.get().getFlowFileActivity().getLatestActivityTime(); + activityTime.ifPresent(time -> { + if (time > latestActivityTime.get()) { + latestActivityTime.set(time); + } + }); + } + + for (final ProcessGroup childGroup : group.getProcessGroups()) { + final OptionalLong activityTime = childGroup.getFlowFileActivity().getLatestActivityTime(); + activityTime.ifPresent(time -> { + if (time > latestActivityTime.get()) { + latestActivityTime.set(time); + } + }); + } + + final long result = latestActivityTime.get(); + return result == -1L ? OptionalLong.empty() : OptionalLong.of(result); + } + + @Override + public FlowFileTransferCounts getTransferCounts() { + long totalReceivedCount = 0L; + long totalReceivedBytes = 0L; + long totalSentCount = 0L; + long totalSentBytes = 0L; + + // Aggregate transfer counts from all processors + for (final Connectable connectable : group.getProcessors()) { + final FlowFileTransferCounts counts = connectable.getFlowFileActivity().getTransferCounts(); + totalReceivedCount += counts.getReceivedCount(); + totalReceivedBytes += counts.getReceivedBytes(); + totalSentCount += counts.getSentCount(); + totalSentBytes += counts.getSentBytes(); + } + + // Aggregate transfer counts from all input ports + for (final Connectable connectable : group.getInputPorts()) { + final FlowFileTransferCounts counts = connectable.getFlowFileActivity().getTransferCounts(); + totalReceivedCount += counts.getReceivedCount(); + totalReceivedBytes += counts.getReceivedBytes(); + totalSentCount += counts.getSentCount(); + totalSentBytes += counts.getSentBytes(); + } + + // Aggregate transfer counts from all output ports + for (final Connectable connectable : group.getOutputPorts()) { + final FlowFileTransferCounts counts = connectable.getFlowFileActivity().getTransferCounts(); + totalReceivedCount += counts.getReceivedCount(); + totalReceivedBytes += counts.getReceivedBytes(); + totalSentCount += counts.getSentCount(); + totalSentBytes += counts.getSentBytes(); + } + + for (final ProcessGroup childGroup : group.getProcessGroups()) { + final FlowFileTransferCounts counts = childGroup.getFlowFileActivity().getTransferCounts(); + totalReceivedCount += counts.getReceivedCount(); + totalReceivedBytes += counts.getReceivedBytes(); + totalSentCount += counts.getSentCount(); + totalSentBytes += counts.getSentBytes(); + } + + // Aggregate transfer counts from stateless group node if present + final Optional statelessGroupNode = group.getStatelessGroupNode(); + if (statelessGroupNode.isPresent()) { + final FlowFileTransferCounts counts = statelessGroupNode.get().getFlowFileActivity().getTransferCounts(); + totalReceivedCount += counts.getReceivedCount(); + totalReceivedBytes += counts.getReceivedBytes(); + totalSentCount += counts.getSentCount(); + totalSentBytes += counts.getSentBytes(); + } + + return new FlowFileTransferCounts(totalReceivedCount, totalReceivedBytes, totalSentCount, totalSentBytes); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java index 490f8f61e295..c506cb0ba727 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java @@ -32,6 +32,7 @@ import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.components.ConnectorMethod; import org.apache.nifi.components.resource.ResourceContext; import org.apache.nifi.components.resource.ResourceReferenceFactory; import org.apache.nifi.components.resource.ResourceReferences; @@ -68,6 +69,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.lang.reflect.Method; import java.net.URL; import java.util.ArrayList; import java.util.Collection; @@ -414,7 +416,7 @@ public void verifyCanUpdateProperties(final Map properties) { } } - protected List verifyConfig(final Map propertyValues, final String annotationData, final ParameterContext parameterContext) { + protected List verifyConfig(final Map propertyValues, final String annotationData, final ParameterLookup parameterLookup) { final List results = new ArrayList<>(); try { @@ -422,7 +424,7 @@ protected List verifyConfig(final Map descriptorToConfigMap = createDescriptorToConfigMap(propertyValues); final ValidationContext validationContext = getValidationContextFactory().newValidationContext(descriptorToConfigMap, annotationData, - getProcessGroupIdentifier(), getIdentifier(), parameterContext, false); + getProcessGroupIdentifier(), getIdentifier(), parameterLookup, false); final ValidationState validationState = performValidation(validationContext); final ValidationStatus validationStatus = validationState.getStatus(); @@ -471,6 +473,22 @@ protected List verifyConfig(final Map propertyValues, final String annotationData, + final ParameterLookup parameterLookup, final boolean validateConnections) { + + final PropertyConfigurationMapper configurationMapper = new PropertyConfigurationMapper(); + final Map descriptorToRawValueMap = new LinkedHashMap<>(); + for (final Map.Entry entry : propertyValues.entrySet()) { + final PropertyDescriptor descriptor = getPropertyDescriptor(entry.getKey()); + final PropertyConfiguration propertyConfiguration = configurationMapper.mapRawPropertyValuesToPropertyConfiguration(descriptor, entry.getValue()); + descriptorToRawValueMap.put(descriptor, propertyConfiguration); + } + + return getValidationContextFactory().newValidationContext(descriptorToRawValueMap, annotationData, getProcessGroupIdentifier(), getIdentifier(), + parameterLookup, validateConnections); + } + private static Map createDescriptorToConfigMap(final Map propertyValues) { final Map descriptorToConfigMap = new LinkedHashMap<>(); for (final Map.Entry entry : propertyValues.entrySet()) { @@ -590,7 +608,9 @@ private void setProperty(final PropertyDescriptor descriptor, final PropertyConf if (!propertyConfiguration.equals(propertyModComparisonValue)) { try { final String oldValue = propertyModComparisonValue == null ? null : propertyModComparisonValue.getEffectiveValue(getParameterContext()); - onPropertyModified(descriptor, oldValue, resolvedValue); + if (!Objects.equals(oldValue, resolvedValue)) { + onPropertyModified(descriptor, oldValue, resolvedValue); + } } catch (final Exception e) { // nothing really to do here... logger.error("Failed to notify {} that property {} changed", this, descriptor, e); @@ -851,13 +871,15 @@ public ValidationState performValidation(final Map results; + final Collection allResults; try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getComponent().getClass(), getIdentifier())) { - results = computeValidationErrors(validationContext); + allResults = computeValidationErrors(validationContext); } - final ValidationStatus status = results.isEmpty() ? ValidationStatus.VALID : ValidationStatus.INVALID; - final ValidationState validationState = new ValidationState(status, results); + final Collection invalidResults = allResults.isEmpty() ? Collections.emptyList() + : allResults.stream().filter(result -> !result.isValid()).toList(); + final ValidationStatus status = invalidResults.isEmpty() ? ValidationStatus.VALID : ValidationStatus.INVALID; + final ValidationState validationState = new ValidationState(status, invalidResults); return validationState; } @@ -876,7 +898,7 @@ public final ValidationStatus performValidation() { } protected Collection computeValidationErrors(final ValidationContext validationContext) { - Throwable failureCause = null; + Throwable failureCause; try { if (!sensitiveDynamicPropertyNames.get().isEmpty() && !isSupportsSensitiveDynamicProperties()) { @@ -910,7 +932,7 @@ protected Collection computeValidationErrors(final ValidationC } final List invalidParameterResults = validateParameterReferences(validationContext); - invalidParameterResults.addAll(validateConfig()); + invalidParameterResults.addAll(validateConfig(validationContext)); if (!invalidParameterResults.isEmpty()) { // At this point, we are not able to properly resolve all property values, so we will not attempt to perform @@ -971,11 +993,12 @@ protected Collection computeValidationErrors(final ValidationC * Validates the current configuration, returning ValidationResults for any * invalid configuration parameter. * + * @param validationContext the ValidationContext to use for parameter lookup * @return Collection of validation result objects for any invalid findings * only. If the collection is empty then the component is valid. Should guarantee * non-null */ - protected abstract List validateConfig(); + protected abstract List validateConfig(ValidationContext validationContext); private List validateParameterReferences(final ValidationContext validationContext) { final List results = new ArrayList<>(); @@ -1539,6 +1562,32 @@ public void verifyCanUpdateBundle(final BundleCoordinate incomingCoordinate) thr } } + protected List getConnectorMethods(final Class componentClass) { + final List connectorMethods = new ArrayList<>(); + for (final Method method : componentClass.getDeclaredMethods()) { + final ConnectorMethod annotation = method.getAnnotation(ConnectorMethod.class); + connectorMethods.add(annotation); + } + + return connectorMethods; + } + + protected Method discoverConnectorMethod(final Class componentClass, final String connectorMethodName) { + for (final Method method : componentClass.getDeclaredMethods()) { + final ConnectorMethod annotation = method.getAnnotation(ConnectorMethod.class); + if (annotation != null && annotation.name().equals(connectorMethodName)) { + return method; + } + } + + final Class superClass = componentClass.getSuperclass(); + if (superClass != null && !Object.class.equals(superClass)) { + return discoverConnectorMethod(superClass, connectorMethodName); + } + + return null; + } + protected void setAdditionalResourcesFingerprint(String additionalResourcesFingerprint) { this.additionalResourcesFingerprint = additionalResourcesFingerprint; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractPort.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractPort.java index 4edb4b8e2ce4..07463946c6a4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractPort.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractPort.java @@ -25,8 +25,10 @@ import org.apache.nifi.components.PortFunction; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.ConnectableFlowFileActivity; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Port; import org.apache.nifi.connectable.Position; import org.apache.nifi.groups.ProcessGroup; @@ -96,6 +98,8 @@ public abstract class AbstractPort implements Port { private final Lock readLock = rwLock.readLock(); private final Lock writeLock = rwLock.writeLock(); + private final ConnectableFlowFileActivity flowFileActivity = new ConnectableFlowFileActivity(); + public AbstractPort(final String id, final String name, final ConnectableType type, final ProcessScheduler scheduler) { this.id = requireNonNull(id); this.name = new AtomicReference<>(requireNonNull(name)); @@ -438,6 +442,7 @@ public void shutdown() { @Override public void onSchedulingStart() { + flowFileActivity.reset(); scheduledState.set(ScheduledState.RUNNING); logger.info("{} started", this); } @@ -737,4 +742,9 @@ public PortFunction getPortFunction() { public void setPortFunction(final PortFunction portFunction) { this.portFunction.set(requireNonNull(portFunction)); } + + @Override + public FlowFileActivity getFlowFileActivity() { + return flowFileActivity; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ComponentNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ComponentNode.java index db9618129971..9461fba3aa9d 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ComponentNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ComponentNode.java @@ -69,6 +69,9 @@ default void setProperties(Map properties) { void verifyCanUpdateProperties(final Map properties); + ValidationContext createValidationContext(Map propertyValues, String annotationData, + ParameterLookup parameterLookup, boolean validateConnections); + /** * @return the Set of names of all Parameters that are referenced by this component */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ParameterProviderNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ParameterProviderNode.java index 04f14b09bb79..a0ce84607c87 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ParameterProviderNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ParameterProviderNode.java @@ -51,6 +51,19 @@ public interface ParameterProviderNode extends ComponentNode { */ void fetchParameters(); + /** + * Fetches all parameter values from the Parameter Provider and returns them. This method does not cache the results for later retrieval. + * @return all Parameter Groups with Parameter Names and Values + */ + List fetchParameterValues(); + + /** + * Fetches parameters that match the provided fully qualified parameter names. This method does not cache the results for later retrieval. + * @param fullyQualifiedParameterNames fully qualified names of parameters to fetch + * @return Parameter Groups with Parameter Names and Values that match the provided names + */ + List fetchParameterValues(List fullyQualifiedParameterNames); + /** * Find a named Parameter Group cached from previous request to fetch Parameters from the configured Parameter Provider * diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessScheduler.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessScheduler.java index 2df3cb477a39..ebef0a18c25e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessScheduler.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessScheduler.java @@ -17,6 +17,7 @@ package org.apache.nifi.controller; import org.apache.nifi.annotation.notification.PrimaryNodeState; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.connectable.Funnel; import org.apache.nifi.connectable.Port; import org.apache.nifi.controller.service.ControllerServiceNode; @@ -60,7 +61,7 @@ public interface ProcessScheduler { * will throw an {@link IllegalStateException}. * @throws IllegalStateException if the Processor is disabled */ - Future startProcessor(ProcessorNode procNode, boolean failIfStopping); + CompletableFuture startProcessor(ProcessorNode procNode, boolean failIfStopping); /** * Starts scheduling the given processor to run once, after invoking all methods @@ -117,7 +118,7 @@ public interface ProcessScheduler { * @param groupNode the group to start * @return a Future that will be completed whenever the group has started */ - Future startStatelessGroup(StatelessGroupNode groupNode); + CompletableFuture startStatelessGroup(StatelessGroupNode groupNode); /** * Stops scheduling the given Stateless Group to run. Returns a Future that will be completed whenever all components within the group have been stopped @@ -251,6 +252,17 @@ public interface ProcessScheduler { */ CompletableFuture enableControllerService(ControllerServiceNode service); + /** + * Enables the Controller Service using the provided ConfigurationContext instead of deriving + * the context from the service's current configuration. This allows enabling a service with + * temporary/override property values. + * + * @param service the controller service to enable + * @param configurationContext the configuration context to use when enabling the service + * @return a CompletableFuture that completes when the service has been enabled + */ + CompletableFuture enableControllerService(ControllerServiceNode service, ConfigurationContext configurationContext); + /** * Disables all of the given Controller Services in the order provided by the List * @param services the controller services to disable @@ -264,6 +276,12 @@ public interface ProcessScheduler { */ CompletableFuture disableControllerService(ControllerServiceNode service); + CompletableFuture startConnector(ConnectorNode connectorNode); + + CompletableFuture stopConnector(ConnectorNode connectorNode); + + void onConnectorRemoved(ConnectorNode connectorNode); + /** * Submits the given task to be executed exactly once in a background thread * diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessorNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessorNode.java index 61a9cc5b5f60..25ce747b7365 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessorNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessorNode.java @@ -20,6 +20,8 @@ import org.apache.nifi.annotation.lifecycle.OnUnscheduled; import org.apache.nifi.annotation.notification.PrimaryNodeState; import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ConnectorMethod; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.components.validation.ValidationTrigger; import org.apache.nifi.connectable.Connectable; @@ -31,6 +33,7 @@ import org.apache.nifi.logging.LogLevel; import org.apache.nifi.migration.ControllerServiceFactory; import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.parameter.ParameterLookup; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.Processor; import org.apache.nifi.processor.Relationship; @@ -136,7 +139,8 @@ public void verifyCanPerformVerification() { } } - public abstract List verifyConfiguration(ProcessContext processContext, ComponentLog logger, Map attributes, ExtensionManager extensionManager); + public abstract List verifyConfiguration(ProcessContext processContext, ComponentLog logger, Map attributes, ExtensionManager extensionManager, + ParameterLookup parameterLookup); public abstract void verifyCanTerminate(); @@ -212,12 +216,12 @@ public abstract void start(ScheduledExecutorService scheduler, long administrati * @param administrativeYieldMillis * the amount of milliseconds to wait for administrative yield * @param timeoutMillis the number of milliseconds to wait after triggering the Processor's @OnScheduled methods before timing out and considering -* the startup a failure. This will result in the thread being interrupted and trying again. + * the startup a failure. This will result in the thread being interrupted and trying again. * @param processContextFactory -* a factory for creating instances of {@link ProcessContext} + * a factory for creating instances of {@link ProcessContext} * @param schedulingAgentCallback -* the callback provided by the {@link ProcessScheduler} to -* execute upon successful start of the Processor + * the callback provided by the {@link ProcessScheduler} to + * execute upon successful start of the Processor */ public abstract void runOnce(ScheduledExecutorService scheduler, long administrativeYieldMillis, long timeoutMillis, Supplier processContextFactory, SchedulingAgentCallback schedulingAgentCallback); @@ -307,4 +311,7 @@ protected void performFlowAnalysisOnThis() { public abstract void migrateConfiguration(Map originalPropertyValues, ControllerServiceFactory serviceFactory); + public abstract String invokeConnectorMethod(String methodName, Map jsonArguments, ProcessContext processContext) throws InvocationFailedException; + + public abstract List getConnectorMethods(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java index b9a0008e5af2..091a3e36433d 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java @@ -24,8 +24,10 @@ import org.apache.nifi.authorization.resource.ResourceType; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.ConnectableFlowFileActivity; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Funnel; import org.apache.nifi.connectable.Position; import org.apache.nifi.flowfile.FlowFile; @@ -84,6 +86,8 @@ public class StandardFunnel implements Funnel { final int maxIterations; private final int maxConcurrentTasks; + private final ConnectableFlowFileActivity flowFileActivity = new ConnectableFlowFileActivity(); + public StandardFunnel(final String identifier, final int maxConcurrentTasks, final int maxBatchSize) { this.identifier = identifier; this.processGroupRef = new AtomicReference<>(); @@ -646,4 +650,9 @@ public void setVersionedComponentId(final String versionedComponentId) { } } } + + @Override + public FlowFileActivity getFlowFileActivity() { + return flowFileActivity; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ValidationContextFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ValidationContextFactory.java index 96806fca1d70..019875c31259 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ValidationContextFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ValidationContextFactory.java @@ -19,7 +19,7 @@ import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.ValidationContext; import org.apache.nifi.controller.flowanalysis.FlowAnalyzer; -import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterLookup; import org.apache.nifi.validation.RuleViolationsManager; import java.util.Map; @@ -27,7 +27,7 @@ public interface ValidationContextFactory { - ValidationContext newValidationContext(Map properties, String annotationData, String groupId, String componentId, ParameterContext parameterContext, + ValidationContext newValidationContext(Map properties, String annotationData, String groupId, String componentId, ParameterLookup parameterLookup, boolean validateConnections); Optional getRuleViolationsManager(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/FlowManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/FlowManager.java index 70c27cd9d8a8..c29788069e50 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/FlowManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/FlowManager.java @@ -17,6 +17,7 @@ package org.apache.nifi.controller.flow; import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Connection; import org.apache.nifi.connectable.Funnel; @@ -164,6 +165,15 @@ public interface FlowManager extends ParameterProviderLookup { */ ProcessGroup getGroup(String id); + /** + * Returns the ProcessGroup with the given ID that is managed by the Connector with the given ID, + * or null if no such ProcessGroup exists. + * @param groupId id of the group + * @param connectorId id of the connector + * @return the ProcessGroup with the given ID or null if none can be found + */ + ProcessGroup getGroup(String groupId, String connectorId); + void onProcessGroupAdded(ProcessGroup group); void onProcessGroupRemoved(ProcessGroup group); @@ -247,7 +257,11 @@ ProcessorNode createProcessor(String type, String id, BundleCoordinate coordinat Port createLocalOutputPort(String id, String name); - ProcessGroup createProcessGroup(String id); + default ProcessGroup createProcessGroup(String id) { + return createProcessGroup(id, null); + } + + ProcessGroup createProcessGroup(String id, String connectorId); void onConnectionAdded(Connection connection); @@ -345,20 +359,32 @@ ControllerServiceNode createControllerService(String type, String id, BundleCoor * IllegalStateException is thrown. See {@link FlowManager#withParameterContextResolution(Runnable)} * for example usage. * - * @param id The unique id - * @param name The ParameterContext name - * @param description The ParameterContext description - * @param parameters The Parameters - * @param inheritedContextIds The identifiers of any Parameter Contexts that the newly created Parameter Context should inherit from. The order of the identifiers in the List determines the + * @param id the unique id + * @param name the ParameterContext name + * @param description the ParameterContext description + * @param parameters the Parameters + * @param inheritedContextIds the identifiers of any Parameter Contexts that the newly created Parameter Context should inherit from. The order of the identifiers in the List determines the * order in which parameters with conflicting names are resolved. I.e., the Parameter Context whose ID comes first in the List is preferred. - * @param parameterProviderConfiguration Optional configuration for a ParameterProvider - * @return The created ParameterContext - * @throws IllegalStateException If parameterContexts is not empty and this method is called without being wrapped + * @param parameterProviderConfiguration optional configuration for a ParameterProvider + * @return the created ParameterContext + * @throws IllegalStateException if parameterContexts is not empty and this method is called without being wrapped * by {@link FlowManager#withParameterContextResolution(Runnable)} */ ParameterContext createParameterContext(String id, String name, String description, Map parameters, List inheritedContextIds, ParameterProviderConfiguration parameterProviderConfiguration); + /** + * Creates an empty Parameter Context with the given ID for the provided root process group. This Parameter Context is not + * registered with the ParameterContextManager. + * + * @param id the id of the new ParameterContext + * @param name the name of the new ParameterContext + * @param description the description of the new ParameterContext + * @param rootGroup the root process group + * @return the duplicated ParameterContext + */ + ParameterContext createEmptyParameterContext(String id, String name, String description, ProcessGroup rootGroup); + /** * Performs the given ParameterContext-related action, and then resolves all inherited ParameterContext references. * Example usage:

@@ -379,6 +405,22 @@ ParameterContext createParameterContext(String id, String name, String descripti ParameterContextManager getParameterContextManager(); + /** + * Creates a connector using hte provided information. + * @param type the fully qualified class name of the connector + * @param id the unique ID of the connector + * @param coordinate the bundle coordinate for this connector + * @param firstTimeAdded whether or not this is the first time this connector is added to the graph. If {@code true}, will invoke methods + * annotated with the {@link org.apache.nifi.annotation.lifecycle.OnAdded} annotation. + * @param registerLogObserver whether or not to register a log observer for this connector + * @return the created connector + */ + ConnectorNode createConnector(String type, String id, BundleCoordinate coordinate, boolean firstTimeAdded, boolean registerLogObserver); + + List getAllConnectors(); + + ConnectorNode getConnector(String id); + /** * @return the number of each type of component (Processor, Controller Service, Process Group, Funnel, Input Port, Output Port, * Parameter Provider, Reporting Task, Flow Analysis Rule, Remote Process Group) diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/VersionedDataflow.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/VersionedDataflow.java index e8b92339a533..83b9464cd1ad 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/VersionedDataflow.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/flow/VersionedDataflow.java @@ -17,6 +17,7 @@ package org.apache.nifi.controller.flow; +import org.apache.nifi.flow.VersionedConnector; import org.apache.nifi.flow.VersionedControllerService; import org.apache.nifi.flow.VersionedFlowAnalysisRule; import org.apache.nifi.flow.VersionedFlowRegistryClient; @@ -36,6 +37,7 @@ public class VersionedDataflow { private List controllerServices; private List reportingTasks; private List flowAnalysisRules; + private List connectors; private VersionedProcessGroup rootGroup; public VersionedFlowEncodingVersion getEncodingVersion() { @@ -102,6 +104,14 @@ public void setParameterProviders(final List paramet this.parameterProviders = parameterProviders; } + public List getConnectors() { + return connectors; + } + + public void setConnectors(final List connectors) { + this.connectors = connectors; + } + public VersionedProcessGroup getRootGroup() { return rootGroup; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileRequest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileRequest.java index 0aec6c750d5a..9400a79b5457 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileRequest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/queue/DropFlowFileRequest.java @@ -17,9 +17,12 @@ package org.apache.nifi.controller.queue; +import java.util.concurrent.CompletableFuture; + public class DropFlowFileRequest implements DropFlowFileStatus { private final String identifier; private final long submissionTime = System.currentTimeMillis(); + private final CompletableFuture completionFuture = new CompletableFuture<>(); private volatile QueueSize originalSize; private volatile QueueSize currentSize; @@ -85,6 +88,11 @@ public String getFailureReason() { return failureReason; } + @Override + public CompletableFuture getCompletionFuture() { + return completionFuture; + } + public synchronized void setState(final DropFlowFileState state) { setState(state, null); } @@ -93,6 +101,12 @@ public synchronized void setState(final DropFlowFileState state, final String ex this.state = state; this.failureReason = explanation; this.lastUpdated = System.currentTimeMillis(); + + if (state == DropFlowFileState.CANCELED || state == DropFlowFileState.COMPLETE) { + completionFuture.complete(null); + } else if (state == DropFlowFileState.FAILURE) { + completionFuture.completeExceptionally(new RuntimeException(explanation)); + } } public synchronized boolean cancel() { @@ -101,6 +115,7 @@ public synchronized boolean cancel() { } this.state = DropFlowFileState.CANCELED; + completionFuture.complete(null); return true; } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceNode.java index 52a4f0db477d..0f1bd09535f9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceNode.java @@ -20,6 +20,8 @@ import org.apache.nifi.components.ConfigVerificationResult; import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.VersionedComponent; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ConnectorMethod; import org.apache.nifi.controller.ComponentNode; import org.apache.nifi.controller.ConfigurationContext; import org.apache.nifi.controller.ControllerService; @@ -30,6 +32,7 @@ import org.apache.nifi.logging.LogLevel; import org.apache.nifi.migration.ControllerServiceFactory; import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.parameter.ParameterLookup; import java.util.List; import java.util.Map; @@ -111,6 +114,20 @@ public interface ControllerServiceNode extends ComponentNode, VersionedComponent */ CompletableFuture enable(ScheduledExecutorService scheduler, long administrativeYieldMillis, boolean completeExceptionallyOnFailure); + /** + * Enables this service using the provided ConfigurationContext, calling any method annotated with @OnEnabled and updating the state of the service. + * This allows enabling a service with temporary/override property values without modifying the service's configuration. + * + * @param scheduler implementation of {@link ScheduledExecutorService} used to initiate service enabling task as well as its retries + * @param administrativeYieldMillis the amount of milliseconds to wait for administrative yield + * @param completeExceptionallyOnFailure if the Controller Service cannot be enabled because it is invalid or throws an Exception from the @OnEnabled lifecycle method, + * dictates whether the CompletableFuture should be completed exceptionally or not + * @param configurationContext the ConfigurationContext to use when enabling the service, or null to use the default + * + * @return a CompletableFuture that can be used to wait for the service to finish enabling + */ + CompletableFuture enable(ScheduledExecutorService scheduler, long administrativeYieldMillis, boolean completeExceptionallyOnFailure, ConfigurationContext configurationContext); + /** * Will disable this service. Disabling of the service typically means * invoking it's operation that is annotated with @OnDisabled. @@ -238,9 +255,11 @@ public interface ControllerServiceNode extends ComponentNode, VersionedComponent * @param logger a logger that can be used when performing verification * @param variables variables that can be used to resolve property values via Expression Language * @param extensionManager extension manager that is used for obtaining appropriate NAR ClassLoaders + * @param parameterLookup the parameter lookup used to resolve parameter references * @return a list of results indicating whether or not the given configuration is valid */ - List verifyConfiguration(ConfigurationContext context, ComponentLog logger, Map variables, ExtensionManager extensionManager); + List verifyConfiguration(ConfigurationContext context, ComponentLog logger, Map variables, ExtensionManager extensionManager, + ParameterLookup parameterLookup); /** * Sets a new proxy and implementation for this node. @@ -257,4 +276,7 @@ void setControllerServiceAndProxy(final LoggableComponent imp void migrateConfiguration(Map originalPropertyValues, ControllerServiceFactory serviceFactory); + String invokeConnectorMethod(String methodName, Map jsonArguments, ConfigurationContext configurationContext) throws InvocationFailedException; + + List getConnectorMethods(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceProvider.java index c836e93063af..6e3755427964 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceProvider.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/service/ControllerServiceProvider.java @@ -84,14 +84,14 @@ public interface ControllerServiceProvider extends ControllerServiceLookup { * @param serviceNodes the nodes * @return a Future that can be used to cancel the task or wait until it is completed */ - Future enableControllerServicesAsync(Collection serviceNodes); + CompletableFuture enableControllerServicesAsync(Collection serviceNodes); /** * Enables the given Controller Service and any dependencies that it has * @param serviceNode the Controller Service to enable * @return a Future that can be used to cancel the task or wait until it is completed */ - Future enableControllerServiceAndDependencies(ControllerServiceNode serviceNode); + CompletableFuture enableControllerServiceAndDependencies(ControllerServiceNode serviceNode); /** * Disables the given controller service so that it cannot be used by other diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/events/BulletinFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/events/BulletinFactory.java index 443ac18e8e73..86c80ce986bf 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/events/BulletinFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/events/BulletinFactory.java @@ -16,6 +16,7 @@ */ package org.apache.nifi.events; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.reporting.Bulletin; @@ -68,6 +69,11 @@ public static Bulletin createBulletin(final Connectable connectable, final Strin return bulletin; } + public static Bulletin createBulletin(final ConnectorNode connector, final String category, final String severity, final String message) { + final String connectorName = connector.getName(); + return createBulletin(null, connectorName, connector.getIdentifier(), ComponentType.CONNECTOR, connector.getName(), category, severity, message, null, null); + } + private static String buildGroupPath(ProcessGroup group) { if (group == null) { return null; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java index cc53fe961dea..91a0a295aa4f 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java @@ -16,10 +16,12 @@ */ package org.apache.nifi.groups; +import org.apache.nifi.authorization.resource.Authorizable; import org.apache.nifi.authorization.resource.ComponentAuthorizable; import org.apache.nifi.components.VersionedComponent; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Funnel; import org.apache.nifi.connectable.Port; import org.apache.nifi.connectable.Positionable; @@ -136,6 +138,11 @@ public interface ProcessGroup extends ComponentAuthorizable, Positionable, Versi */ void setName(String name); + /** + * @return the ID of the Connector that is responsible for this Process Group, or an empty optional if no Connector is associated + */ + Optional getConnectorIdentifier(); + /** * @return the user-set comments about this ProcessGroup, or * null if no comments have been set @@ -255,7 +262,7 @@ default CompletableFuture stopComponents() { * @throws IllegalStateException if the processor is not valid, or is * already running */ - Future startProcessor(ProcessorNode processor, boolean failIfStopping); + CompletableFuture startProcessor(ProcessorNode processor, boolean failIfStopping); /** * Runs the given Processor once and the stops it by calling the provided callback. @@ -1268,4 +1275,26 @@ default CompletableFuture stopComponents() { * @return the configured maximum amount of time that a Stateless Flow can run before it times out and is considered a failure */ String getStatelessFlowTimeout(); + + /** + * @return the FlowFileActivity for this Process Group + */ + FlowFileActivity getFlowFileActivity(); + + /** + * Sets an explicit Authorizable that is to be used as the Process Group's parent for the purposes of authorization. + * If not set, the Process Group's parent group will be used. + * @param parent the parent Authorizable to set, or null to clear any explicit parent + */ + void setExplicitParentAuthorizable(Authorizable parent); + + /** + * Deletes any data in any FlowFile Queues in this Process Group and any child Process Groups + * and removes any components (Controller Services, Processors, Connections, etc.) from this Process Group + * and any child Process Groups. Because the emptying of FlowFile Queues may be a long-running operation, + * this method returns a CompletableFuture that will be completed once the purge operation has finished. + * + * @return a CompletableFuture that will be completed once the purge operation has finished + */ + CompletableFuture purge(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/RemoteProcessGroup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/RemoteProcessGroup.java index 87676aae9372..201200b4ed96 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/RemoteProcessGroup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/RemoteProcessGroup.java @@ -30,7 +30,7 @@ import java.util.Collection; import java.util.Date; import java.util.Set; -import java.util.concurrent.Future; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.TimeUnit; public interface RemoteProcessGroup extends ComponentAuthorizable, Positionable, VersionedComponent { @@ -127,7 +127,7 @@ public interface RemoteProcessGroup extends ComponentAuthorizable, Positionable, * Immediately terminates communications between this instance and the * remote instance. */ - Future stopTransmitting(); + CompletableFuture stopTransmitting(); /** * Initiates communications between this instance and the remote instance diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/manifest/RuntimeManifestService.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/manifest/RuntimeManifestService.java index 3acbfda5b4d9..60c9463cbd06 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/manifest/RuntimeManifestService.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/manifest/RuntimeManifestService.java @@ -43,12 +43,24 @@ public interface RuntimeManifestService { RuntimeManifest getManifestForBundle(String group, String artifact, String version); /** - * Returns a mapping of additionalDetails for the speicfied bundle. + * Returns a mapping of additionalDetails for the specified bundle. * * @param group The bundle group * @param artifact The bundle artifact * @param version The bundle version - * @return The additionaDetails mapping + * @return The additionalDetails mapping */ Map discoverAdditionalDetails(String group, String artifact, String version); + + /** + * Returns a mapping of step documentation files for the specified Connector type. + * The keys are the step names and the values are the corresponding documentation files. + * + * @param group The bundle group + * @param artifact The bundle artifact + * @param version The bundle version + * @param connectorType The fully qualified class name of the Connector + * @return The step documentation mapping + */ + Map discoverStepDocumentation(String group, String artifact, String version, String connectorType); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/connectable/ProcessGroupFlowFileActivityTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/connectable/ProcessGroupFlowFileActivityTest.java new file mode 100644 index 000000000000..754b955dad03 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/connectable/ProcessGroupFlowFileActivityTest.java @@ -0,0 +1,162 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectable; + +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.groups.StatelessGroupNode; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.OptionalLong; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class ProcessGroupFlowFileActivityTest { + + @Mock + private ProcessGroup processGroup; + + private ProcessGroupFlowFileActivity activity; + + @BeforeEach + void setUp() { + activity = new ProcessGroupFlowFileActivity(processGroup); + + lenient().when(processGroup.getProcessors()).thenReturn(Collections.emptyList()); + lenient().when(processGroup.getInputPorts()).thenReturn(Collections.emptySet()); + lenient().when(processGroup.getOutputPorts()).thenReturn(Collections.emptySet()); + lenient().when(processGroup.getFunnels()).thenReturn(Collections.emptySet()); + lenient().when(processGroup.getProcessGroups()).thenReturn(Collections.emptySet()); + lenient().when(processGroup.getStatelessGroupNode()).thenReturn(Optional.empty()); + } + + @Test + void testGetTransferCountsWithNoComponents() { + final FlowFileTransferCounts counts = activity.getTransferCounts(); + + assertEquals(0L, counts.getReceivedCount()); + assertEquals(0L, counts.getReceivedBytes()); + assertEquals(0L, counts.getSentCount()); + assertEquals(0L, counts.getSentBytes()); + } + + @Test + void testGetTransferCountsAggregatesProcessors() { + final ProcessorNode processor = createMockConnectable(ProcessorNode.class, 10, 100L, 5, 50L); + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + + final FlowFileTransferCounts counts = activity.getTransferCounts(); + + assertEquals(10L, counts.getReceivedCount()); + assertEquals(100L, counts.getReceivedBytes()); + assertEquals(5L, counts.getSentCount()); + assertEquals(50L, counts.getSentBytes()); + } + + @Test + void testGetTransferCountsIncludesStatelessGroupNode() { + final StatelessGroupNode statelessGroupNode = mock(StatelessGroupNode.class); + final FlowFileActivity statelessActivity = new ConnectableFlowFileActivity(); + statelessActivity.updateTransferCounts(20, 2000L, 15, 1500L); + + when(statelessGroupNode.getFlowFileActivity()).thenReturn(statelessActivity); + when(processGroup.getStatelessGroupNode()).thenReturn(Optional.of(statelessGroupNode)); + + final FlowFileTransferCounts counts = activity.getTransferCounts(); + + assertEquals(20L, counts.getReceivedCount()); + assertEquals(2000L, counts.getReceivedBytes()); + assertEquals(15L, counts.getSentCount()); + assertEquals(1500L, counts.getSentBytes()); + } + + @Test + void testGetTransferCountsAggregatesProcessorsAndStatelessGroupNode() { + final ProcessorNode processor = createMockConnectable(ProcessorNode.class, 10, 100L, 5, 50L); + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + + final StatelessGroupNode statelessGroupNode = mock(StatelessGroupNode.class); + final FlowFileActivity statelessActivity = new ConnectableFlowFileActivity(); + statelessActivity.updateTransferCounts(20, 2000L, 15, 1500L); + when(statelessGroupNode.getFlowFileActivity()).thenReturn(statelessActivity); + when(processGroup.getStatelessGroupNode()).thenReturn(Optional.of(statelessGroupNode)); + + final FlowFileTransferCounts counts = activity.getTransferCounts(); + + assertEquals(30L, counts.getReceivedCount()); + assertEquals(2100L, counts.getReceivedBytes()); + assertEquals(20L, counts.getSentCount()); + assertEquals(1550L, counts.getSentBytes()); + } + + @Test + void testGetTransferCountsAggregatesChildGroups() { + final ProcessGroup childGroup = mock(ProcessGroup.class); + final FlowFileActivity childActivity = mock(FlowFileActivity.class); + when(childActivity.getTransferCounts()).thenReturn(new FlowFileTransferCounts(8, 800L, 3, 300L)); + when(childGroup.getFlowFileActivity()).thenReturn(childActivity); + when(processGroup.getProcessGroups()).thenReturn(Set.of(childGroup)); + + final FlowFileTransferCounts counts = activity.getTransferCounts(); + + assertEquals(8L, counts.getReceivedCount()); + assertEquals(800L, counts.getReceivedBytes()); + assertEquals(3L, counts.getSentCount()); + assertEquals(300L, counts.getSentBytes()); + } + + @Test + void testGetLatestActivityTimeEmpty() { + final OptionalLong result = activity.getLatestActivityTime(); + assertTrue(result.isEmpty()); + } + + @Test + void testGetLatestActivityTimeIncludesStatelessGroupNode() { + final StatelessGroupNode statelessGroupNode = mock(StatelessGroupNode.class); + final ConnectableFlowFileActivity statelessActivity = new ConnectableFlowFileActivity(); + statelessActivity.updateLatestActivityTime(); + when(statelessGroupNode.getFlowFileActivity()).thenReturn(statelessActivity); + when(processGroup.getStatelessGroupNode()).thenReturn(Optional.of(statelessGroupNode)); + + final OptionalLong result = activity.getLatestActivityTime(); + assertTrue(result.isPresent()); + } + + private T createMockConnectable(final Class type, + final int receivedCount, final long receivedBytes, final int sentCount, final long sentBytes) { + final T connectable = mock(type); + final FlowFileActivity connectableActivity = new ConnectableFlowFileActivity(); + connectableActivity.updateTransferCounts(receivedCount, receivedBytes, sentCount, sentBytes); + when(connectable.getFlowFileActivity()).thenReturn(connectableActivity); + return connectable; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/controller/TestAbstractComponentNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/controller/TestAbstractComponentNode.java index e9d383c60df2..8f8edd4724b1 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/controller/TestAbstractComponentNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/test/java/org/apache/nifi/controller/TestAbstractComponentNode.java @@ -536,7 +536,7 @@ public Resource getResource() { } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml b/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml index 0a3a60621e05..351d3c33fecd 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml @@ -128,7 +128,6 @@ org.apache.nifi nifi-expression-language 2.9.0-SNAPSHOT - test org.apache.nifi @@ -290,6 +289,12 @@ netty-transport test + + org.apache.nifi + nifi-connector-utils + 2.9.0-SNAPSHOT + test + com.github.ben-manes.caffeine caffeine @@ -319,6 +324,9 @@ src/test/resources/old-swap-file.swap src/test/resources/xxe_template.xml src/test/resources/swap/444-old-swap-file.swap + src/test/resources/colors.txt + src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Another_Test_Step.md + src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Test_Step.md diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/AssetsRestApiClient.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/AssetsRestApiClient.java index b8f2052dd514..29cecc02a0e4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/AssetsRestApiClient.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/AssetsRestApiClient.java @@ -40,13 +40,14 @@ class AssetsRestApiClient extends NiFiRestApiClient { private static final String NIFI_API_PATH_PART = "nifi-api"; private static final String PARAMETER_CONTEXTS_PATH_PART = "parameter-contexts"; + private static final String CONNECTORS_PATH_PART = "connectors"; private static final String ASSETS_PATH_PART = "assets"; public AssetsRestApiClient(final WebClientService webClientService, final String host, final int port, final boolean secure) { super(webClientService, host, port, secure); } - public AssetsEntity getAssets(final String parameterContextId) { + public AssetsEntity getParameterContextAssets(final String parameterContextId) { final URI requestUri = new StandardHttpUriBuilder() .scheme(baseUri.getScheme()) .host(baseUri.getHost()) @@ -56,7 +57,7 @@ public AssetsEntity getAssets(final String parameterContextId) { .addPathSegment(parameterContextId) .addPathSegment(ASSETS_PATH_PART) .build(); - logger.debug("Requesting Asset listing from {}", requestUri); + logger.debug("Requesting Parameter Context Asset listing from {}", requestUri); // Send the replicated header so that the cluster coordinator does not replicate the request, otherwise this call can happen when no nodes // are considered connected and result in a 500 exception that can't easily be differentiated from other unknown errors @@ -68,7 +69,7 @@ public AssetsEntity getAssets(final String parameterContextId) { return executeEntityRequest(requestUri, requestBodySpec, AssetsEntity.class); } - public InputStream getAssetContent(final String parameterContextId, final String assetId) { + public InputStream getParameterContextAssetContent(final String parameterContextId, final String assetId) { final URI requestUri = new StandardHttpUriBuilder() .scheme(baseUri.getScheme()) .host(baseUri.getHost()) @@ -79,7 +80,53 @@ public InputStream getAssetContent(final String parameterContextId, final String .addPathSegment(ASSETS_PATH_PART) .addPathSegment(assetId) .build(); - logger.debug("Getting asset content from {}", requestUri); + logger.debug("Getting Parameter Context asset content from {}", requestUri); + + try { + final HttpResponseEntity response = webClientService.get() + .uri(requestUri) + .header(ACCEPT_HEADER, APPLICATION_OCTET_STREAM) + .retrieve(); + return getResponseBody(requestUri, response); + } catch (final WebClientServiceException e) { + throw new NiFiRestApiRetryableException(e.getMessage(), e); + } + } + + public AssetsEntity getConnectorAssets(final String connectorId) { + final URI requestUri = new StandardHttpUriBuilder() + .scheme(baseUri.getScheme()) + .host(baseUri.getHost()) + .port(baseUri.getPort()) + .addPathSegment(NIFI_API_PATH_PART) + .addPathSegment(CONNECTORS_PATH_PART) + .addPathSegment(connectorId) + .addPathSegment(ASSETS_PATH_PART) + .build(); + logger.debug("Requesting Connector Asset listing from {}", requestUri); + + // Send the replicated header so that the cluster coordinator does not replicate the request, otherwise this call can happen when no nodes + // are considered connected and result in a 500 exception that can't easily be differentiated from other unknown errors + final HttpRequestBodySpec requestBodySpec = webClientService.get() + .uri(requestUri) + .header(ACCEPT_HEADER, APPLICATION_JSON) + .header(REQUEST_REPLICATED_HEADER, Boolean.TRUE.toString()); + + return executeEntityRequest(requestUri, requestBodySpec, AssetsEntity.class); + } + + public InputStream getConnectorAssetContent(final String connectorId, final String assetId) { + final URI requestUri = new StandardHttpUriBuilder() + .scheme(baseUri.getScheme()) + .host(baseUri.getHost()) + .port(baseUri.getPort()) + .addPathSegment(NIFI_API_PATH_PART) + .addPathSegment(CONNECTORS_PATH_PART) + .addPathSegment(connectorId) + .addPathSegment(ASSETS_PATH_PART) + .addPathSegment(assetId) + .build(); + logger.debug("Getting Connector asset content from {}", requestUri); try { final HttpResponseEntity response = webClientService.get() diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetComponentManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetComponentManager.java index 9954c947c154..9449ce8ec6d4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetComponentManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetComponentManager.java @@ -58,7 +58,7 @@ public StandardAssetComponentManager(final FlowController flowController) { @Override public void restartReferencingComponentsAsync(final Asset asset) { final String assetName = asset.getName(); - final String paramContextId = asset.getParameterContextIdentifier(); + final String paramContextId = asset.getOwnerIdentifier(); Thread.ofVirtual().name("Restart Components for Asset [%s] in ParameterContext [%s]".formatted(assetName, paramContextId)).start(() -> { try { restartReferencingComponents(asset); @@ -71,7 +71,7 @@ public void restartReferencingComponentsAsync(final Asset asset) { @Override public void restartReferencingComponents(final Asset asset) { final ParameterContextManager parameterContextManager = flowManager.getParameterContextManager(); - final ParameterContext parameterContext = parameterContextManager.getParameterContext(asset.getParameterContextIdentifier()); + final ParameterContext parameterContext = parameterContextManager.getParameterContext(asset.getOwnerIdentifier()); // Determine which parameters reference the given asset final Set parametersReferencingAsset = getParametersReferencingAsset(parameterContext, asset); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetSynchronizer.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetSynchronizer.java index 96b4f027cf01..c4bd08fbd372 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetSynchronizer.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardAssetSynchronizer.java @@ -212,7 +212,7 @@ private AssetsEntity listAssetsWithRetry(final AssetsRestApiClient assetsRestApi private AssetsEntity listAssets(final AssetsRestApiClient assetsRestApiClient, final String parameterContextId) { try { - return assetsRestApiClient.getAssets(parameterContextId); + return assetsRestApiClient.getParameterContextAssets(parameterContextId); } catch (final NiFiRestApiRetryableException e) { final Throwable rootCause = ExceptionUtils.getRootCause(e); logger.warn("{}, root cause [{}]: retrying", e.getMessage(), rootCause.getMessage()); @@ -242,7 +242,7 @@ private Asset synchronizeAsset(final AssetsRestApiClient assetsRestApiClient, fi if (Boolean.TRUE == coordinatorAsset.getMissingContent()) { return assetManager.createMissingAsset(parameterContextId, assetName); } else { - try (final InputStream assetInputStream = assetsRestApiClient.getAssetContent(parameterContextId, assetId)) { + try (final InputStream assetInputStream = assetsRestApiClient.getParameterContextAssetContent(parameterContextId, assetId)) { return assetManager.createAsset(parameterContextId, assetName, assetInputStream); } catch (final NiFiRestApiRetryableException e) { final Throwable rootCause = ExceptionUtils.getRootCause(e); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardConnectorAssetSynchronizer.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardConnectorAssetSynchronizer.java new file mode 100644 index 000000000000..ad70446ce5fc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardConnectorAssetSynchronizer.java @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.asset; + +import org.apache.commons.lang3.exception.ExceptionUtils; +import org.apache.nifi.client.NiFiRestApiRetryableException; +import org.apache.nifi.cluster.coordination.ClusterCoordinator; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.web.api.dto.AssetDTO; +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.AssetsEntity; +import org.apache.nifi.web.client.api.WebClientService; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.InputStream; +import java.time.Duration; +import java.time.Instant; +import java.util.Collection; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; +import java.util.function.Function; +import java.util.stream.Collectors; + +/** + * Synchronizes connector assets from the cluster coordinator using the NiFi REST API. + */ +public class StandardConnectorAssetSynchronizer implements AssetSynchronizer { + + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetSynchronizer.class); + + private static final Duration CLUSTER_COORDINATOR_RETRY_DURATION = Duration.ofSeconds(60); + private static final Duration LIST_ASSETS_RETRY_DURATION = Duration.ofMinutes(5); + private static final Duration SYNC_ASSET_RETRY_DURATION = Duration.ofSeconds(30); + + private final AssetManager assetManager; + private final FlowManager flowManager; + private final ConnectorRepository connectorRepository; + private final ClusterCoordinator clusterCoordinator; + private final WebClientService webClientService; + private final NiFiProperties properties; + + public StandardConnectorAssetSynchronizer(final FlowController flowController, + final ClusterCoordinator clusterCoordinator, + final WebClientService webClientService, + final NiFiProperties properties) { + this.assetManager = flowController.getConnectorAssetManager(); + this.flowManager = flowController.getFlowManager(); + this.connectorRepository = flowController.getConnectorRepository(); + this.clusterCoordinator = clusterCoordinator; + this.webClientService = webClientService; + this.properties = properties; + } + + @Override + public void synchronize() { + if (clusterCoordinator == null) { + logger.info("Clustering is not configured: Connector asset synchronization disabled"); + return; + } + + final NodeIdentifier coordinatorNodeId = getElectedActiveCoordinatorNode(); + if (coordinatorNodeId == null) { + logger.warn("Unable to obtain the node identifier for the cluster coordinator: Connector asset synchronization disabled"); + return; + } + + if (clusterCoordinator.isActiveClusterCoordinator()) { + logger.info("Current node is the cluster coordinator: Connector asset synchronization disabled"); + return; + } + + final String coordinatorAddress = coordinatorNodeId.getApiAddress(); + final int coordinatorPort = coordinatorNodeId.getApiPort(); + final AssetsRestApiClient assetsRestApiClient = new AssetsRestApiClient(webClientService, coordinatorAddress, coordinatorPort, properties.isHTTPSConfigured()); + logger.info("Synchronizing connector assets with cluster coordinator at {}:{}", coordinatorAddress, coordinatorPort); + + final List connectors = flowManager.getAllConnectors(); + logger.info("Found {} connectors for synchronizing assets", connectors.size()); + + final Set connectorsWithSynchronizedAssets = new HashSet<>(); + for (final ConnectorNode connector : connectors) { + try { + final boolean assetSynchronized = synchronize(assetsRestApiClient, connector); + if (assetSynchronized) { + connectorsWithSynchronizedAssets.add(connector); + } + } catch (final Exception e) { + logger.error("Failed to synchronize assets for connector [{}]", connector.getIdentifier(), e); + } + } + + restartConnectorsWithSynchronizedAssets(connectorsWithSynchronizedAssets); + } + + private void restartConnectorsWithSynchronizedAssets(final Set connectorsWithSynchronizedAssets) { + for (final ConnectorNode connector : connectorsWithSynchronizedAssets) { + final ConnectorState currentState = connector.getDesiredState(); + if (currentState == ConnectorState.RUNNING) { + logger.info("Restarting connector [{}] after asset synchronization", connector.getIdentifier()); + try { + final Future restartFuture = connectorRepository.restartConnector(connector); + restartFuture.get(); + logger.info("Successfully restarted connector [{}] after asset synchronization", connector.getIdentifier()); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + logger.error("Interrupted while restarting connector [{}] after asset synchronization", connector.getIdentifier(), e); + } catch (final ExecutionException e) { + logger.error("Failed to restart connector [{}] after asset synchronization", connector.getIdentifier(), e.getCause()); + } + } else { + logger.info("Connector [{}] is not running (state={}): skipping restart after asset synchronization", connector.getIdentifier(), currentState); + } + } + } + + private boolean synchronize(final AssetsRestApiClient assetsRestApiClient, final ConnectorNode connector) { + final String connectorId = connector.getIdentifier(); + final Map existingAssets = assetManager.getAssets(connectorId).stream() + .collect(Collectors.toMap(Asset::getIdentifier, Function.identity())); + + final AssetsEntity coordinatorAssetsEntity = listConnectorAssetsWithRetry(assetsRestApiClient, connectorId); + if (coordinatorAssetsEntity == null) { + logger.error("Timeout listing assets from cluster coordinator for connector [{}]", connectorId); + return false; + } + + final Collection coordinatorAssets = coordinatorAssetsEntity.getAssets(); + if (coordinatorAssets == null || coordinatorAssets.isEmpty()) { + logger.info("Connector [{}] did not return any assets from the cluster coordinator", connectorId); + return false; + } + + logger.info("Connector [{}] returned {} assets from the cluster coordinator", connectorId, coordinatorAssets.size()); + + boolean assetSynchronized = false; + for (final AssetEntity coordinatorAssetEntity : coordinatorAssets) { + final AssetDTO coordinatorAsset = coordinatorAssetEntity.getAsset(); + final Asset matchingAsset = existingAssets.get(coordinatorAsset.getId()); + try { + final boolean assetWasSynchronized = synchronize(assetsRestApiClient, connectorId, coordinatorAsset, matchingAsset); + if (assetWasSynchronized) { + assetSynchronized = true; + } + } catch (final Exception e) { + logger.error("Failed to synchronize asset [id={},name={}] for connector [{}]", + coordinatorAsset.getId(), coordinatorAsset.getName(), connectorId, e); + } + } + return assetSynchronized; + } + + private boolean synchronize(final AssetsRestApiClient assetsRestApiClient, final String connectorId, final AssetDTO coordinatorAsset, final Asset matchingAsset) { + final String assetId = coordinatorAsset.getId(); + final String assetName = coordinatorAsset.getName(); + if (matchingAsset == null || !matchingAsset.getFile().exists()) { + logger.info("Synchronizing missing asset [id={},name={}] for connector [{}]", assetId, assetName, connectorId); + synchronizeConnectorAssetWithRetry(assetsRestApiClient, connectorId, coordinatorAsset); + return true; + } else { + final String coordinatorAssetDigest = coordinatorAsset.getDigest(); + final String matchingAssetDigest = matchingAsset.getDigest().orElse(null); + if (!coordinatorAssetDigest.equals(matchingAssetDigest)) { + logger.info("Synchronizing asset [id={},name={}] with updated digest [{}] for connector [{}]", + assetId, assetName, coordinatorAssetDigest, connectorId); + synchronizeConnectorAssetWithRetry(assetsRestApiClient, connectorId, coordinatorAsset); + return true; + } else { + logger.info("Coordinator asset [id={},name={}] found for connector [{}]: retrieval not required", assetId, assetName, connectorId); + return false; + } + } + } + + private AssetsEntity listConnectorAssetsWithRetry(final AssetsRestApiClient assetsRestApiClient, final String connectorId) { + final Instant expirationTime = Instant.ofEpochMilli(System.currentTimeMillis() + LIST_ASSETS_RETRY_DURATION.toMillis()); + while (System.currentTimeMillis() < expirationTime.toEpochMilli()) { + final AssetsEntity assetsEntity = listConnectorAssets(assetsRestApiClient, connectorId); + if (assetsEntity != null) { + return assetsEntity; + } + logger.info("Unable to list assets from cluster coordinator for connector [{}]: retrying until [{}]", connectorId, expirationTime); + sleep(Duration.ofSeconds(5)); + } + return null; + } + + private AssetsEntity listConnectorAssets(final AssetsRestApiClient assetsRestApiClient, final String connectorId) { + try { + return assetsRestApiClient.getConnectorAssets(connectorId); + } catch (final NiFiRestApiRetryableException e) { + final Throwable rootCause = ExceptionUtils.getRootCause(e); + logger.warn("{}, root cause [{}]: retrying", e.getMessage(), rootCause.getMessage()); + return null; + } + } + + private void synchronizeConnectorAssetWithRetry(final AssetsRestApiClient assetsRestApiClient, final String connectorId, final AssetDTO coordinatorAsset) { + final Instant expirationTime = Instant.ofEpochMilli(System.currentTimeMillis() + SYNC_ASSET_RETRY_DURATION.toMillis()); + while (System.currentTimeMillis() < expirationTime.toEpochMilli()) { + final Asset syncedAsset = synchronizeConnectorAsset(assetsRestApiClient, connectorId, coordinatorAsset); + if (syncedAsset != null) { + return; + } + logger.info("Unable to synchronize asset [id={},name={}] for connector [{}]: retrying until [{}]", + coordinatorAsset.getId(), coordinatorAsset.getName(), connectorId, expirationTime); + sleep(Duration.ofSeconds(5)); + } + } + + private Asset synchronizeConnectorAsset(final AssetsRestApiClient assetsRestApiClient, final String connectorId, final AssetDTO coordinatorAsset) { + final String assetId = coordinatorAsset.getId(); + final String assetName = coordinatorAsset.getName(); + if (Boolean.TRUE == coordinatorAsset.getMissingContent()) { + return assetManager.createMissingAsset(connectorId, assetName); + } else { + try (final InputStream assetInputStream = assetsRestApiClient.getConnectorAssetContent(connectorId, assetId)) { + return assetManager.saveAsset(connectorId, assetId, assetName, assetInputStream); + } catch (final NiFiRestApiRetryableException e) { + final Throwable rootCause = ExceptionUtils.getRootCause(e); + logger.warn("{}, root cause [{}]: retrying", e.getMessage(), rootCause.getMessage()); + return null; + } catch (final IOException e) { + final Throwable rootCause = ExceptionUtils.getRootCause(e); + logger.warn("Asset Manager failed to create asset [id={},name={}], root cause [{}]: retrying", assetId, assetId, rootCause.getMessage()); + return null; + } + } + } + + private NodeIdentifier getElectedActiveCoordinatorNode() { + final Instant expirationTime = Instant.ofEpochMilli(System.currentTimeMillis() + CLUSTER_COORDINATOR_RETRY_DURATION.toMillis()); + while (System.currentTimeMillis() < expirationTime.toEpochMilli()) { + final NodeIdentifier coordinatorNodeId = clusterCoordinator.getElectedActiveCoordinatorNode(); + if (coordinatorNodeId != null) { + return coordinatorNodeId; + } + logger.info("Node identifier for the active cluster coordinator is not known yet: retrying until [{}]", expirationTime); + sleep(Duration.ofSeconds(2)); + } + return null; + } + + private void sleep(final Duration duration) { + try { + Thread.sleep(duration); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorDetails.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorDetails.java new file mode 100644 index 000000000000..4ffafd865238 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorDetails.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.logging.ComponentLog; + +public class ConnectorDetails { + private final Connector connector; + private final ComponentLog componentLog; + private final BundleCoordinate bundleCoordinate; + + public ConnectorDetails(final Connector connector, final BundleCoordinate bundleCoordinate, final ComponentLog logger) { + this.connector = connector; + this.bundleCoordinate = bundleCoordinate; + this.componentLog = logger; + } + + public Connector getConnector() { + return connector; + } + + public ComponentLog getComponentLog() { + return componentLog; + } + + public BundleCoordinate getBundleCoordinate() { + return bundleCoordinate; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorParameterLookup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorParameterLookup.java new file mode 100644 index 000000000000..ada91a1d418f --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorParameterLookup.java @@ -0,0 +1,193 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.components.ParameterValue; +import org.apache.nifi.flow.VersionedAsset; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterLookup; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +public class ConnectorParameterLookup implements ParameterLookup { + private final Map parameters; + private final List parameterValues; + private final AssetManager assetManager; + + // TODO: Refactor how parameters work + // It is possible that a VersionedProcessGroup has a child where different parameter contexts are used or are used in a different inheritance order. + public ConnectorParameterLookup(final Collection parameterContexts, final AssetManager assetManager) { + this.assetManager = assetManager; + + parameterValues = Collections.unmodifiableList(createParameterValues(parameterContexts)); + parameters = new HashMap<>(); + for (final ParameterValue parameterValue : parameterValues) { + parameters.put(parameterValue.getName(), createParameter(parameterValue)); + } + } + + private Parameter createParameter(final ParameterValue parameterValue) { + return new Parameter.Builder() + .name(parameterValue.getName()) + .value(parameterValue.getValue()) + .sensitive(parameterValue.isSensitive()) + .referencedAssets(parameterValue.getAssets()) + .build(); + } + + public VersionedParameterContext createVersionedParameterContext(final String name) { + final VersionedParameterContext context = new VersionedParameterContext(); + context.setName(name); + + final Set versionedParameters = new HashSet<>(); + for (final ParameterValue parameterValue : parameterValues) { + final VersionedParameter versionedParameter = new VersionedParameter(); + versionedParameter.setName(parameterValue.getName()); + versionedParameter.setValue(parameterValue.getValue()); + versionedParameter.setSensitive(parameterValue.isSensitive()); + versionedParameters.add(versionedParameter); + } + + context.setParameters(versionedParameters); + return context; + } + + @Override + public Optional getParameter(final String parameterName) { + return Optional.ofNullable(parameters.get(parameterName)); + } + + @Override + public boolean isEmpty() { + return parameters.isEmpty(); + } + + @Override + public long getVersion() { + return 0; + } + + public List getParameterValues() { + return parameterValues; + } + + /** + * Converts a {@code List} found in a VersionedExternalFlow to a + * {@code List} that can be used to update a ParameterContext from a Connector, + * respecting parameter context inheritance and precedence. + * + * @param parameterContexts the list of parameter contexts from a VersionedExternalFlow + * @return the list of ParameterValues + */ + public List createParameterValues(final Collection parameterContexts) { + final List parameterValues = new ArrayList<>(); + + if (parameterContexts == null || parameterContexts.isEmpty()) { + return parameterValues; + } + + // Create a map for easy lookup of parameter contexts by name + final Map contextMap = new HashMap<>(); + for (final VersionedParameterContext context : parameterContexts) { + contextMap.put(context.getName(), context); + } + + // Process each parameter context, including inherited contexts + final Set processedContexts = new HashSet<>(); + for (final VersionedParameterContext context : parameterContexts) { + collectParameterValues(context, contextMap, processedContexts, parameterValues); + } + + return parameterValues; + } + + private void collectParameterValues(final VersionedParameterContext context, final Map contextMap, + final Set processedContexts, final List parameterValues) { + + if (context == null || processedContexts.contains(context.getName())) { + return; + } + + processedContexts.add(context.getName()); + + // Create a map to track existing parameters for efficient lookup + final Map existingParametersByName = new HashMap<>(); + for (final ParameterValue existing : parameterValues) { + existingParametersByName.put(existing.getName(), existing); + } + + // First, process inherited parameter contexts in reverse order (lowest precedence first) + // This ensures that the first inherited context (highest precedence) will override later ones + if (context.getInheritedParameterContexts() != null && !context.getInheritedParameterContexts().isEmpty()) { + final List inheritedContextNames = context.getInheritedParameterContexts(); + // Process in reverse order so that the first (highest precedence) inherited context processes last + for (int i = inheritedContextNames.size() - 1; i >= 0; i--) { + final String inheritedContextName = inheritedContextNames.get(i); + final VersionedParameterContext inheritedContext = contextMap.get(inheritedContextName); + if (inheritedContext != null) { + collectParameterValues(inheritedContext, contextMap, processedContexts, parameterValues); + } + } + } + + // Then, process this context's own parameters (they have the highest precedence and override all inherited ones) + if (context.getParameters() != null) { + // Rebuild the existing parameters map since inherited contexts may have added parameters + existingParametersByName.clear(); + for (final ParameterValue existing : parameterValues) { + existingParametersByName.put(existing.getName(), existing); + } + + for (final VersionedParameter versionedParameter : context.getParameters()) { + final String parameterName = versionedParameter.getName(); + + // Remove existing parameter if present, then add the new one (current context overrides) + if (existingParametersByName.containsKey(parameterName)) { + parameterValues.removeIf(param -> param.getName().equals(parameterName)); + } + + final ParameterValue.Builder builder = new ParameterValue.Builder() + .name(parameterName) + .value(versionedParameter.getValue()) + .sensitive(versionedParameter.isSensitive()); + + if (assetManager != null && versionedParameter.getReferencedAssets() != null) { + for (final VersionedAsset versionedAsset : versionedParameter.getReferencedAssets()) { + assetManager.getAsset(versionedAsset.getIdentifier()).ifPresent(builder::addReferencedAsset); + } + } + + final ParameterValue parameterValue = builder.build(); + + parameterValues.add(parameterValue); + } + } + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorValidationContextBridge.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorValidationContextBridge.java new file mode 100644 index 000000000000..6f867925be7c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/ConnectorValidationContextBridge.java @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.attribute.expression.language.StandardPropertyValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.controller.ControllerService; +import org.apache.nifi.controller.ControllerServiceLookup; +import org.apache.nifi.documentation.init.EmptyControllerServiceLookup; +import org.apache.nifi.expression.ExpressionLanguageCompiler; +import org.apache.nifi.parameter.ExpressionLanguageAgnosticParameterParser; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterLookup; +import org.apache.nifi.parameter.ParameterParser; +import org.apache.nifi.parameter.ParameterTokenList; + +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +public class ConnectorValidationContextBridge implements ValidationContext { + private final Map rawValues; + private final ParameterLookup parameterLookup; + + public ConnectorValidationContextBridge(final Map rawValues, final ParameterLookup parameterLookup) { + this.rawValues = rawValues; + this.parameterLookup = parameterLookup; + } + + @Override + public ControllerServiceLookup getControllerServiceLookup() { + return new EmptyControllerServiceLookup(); + } + + @Override + public ValidationContext getControllerServiceValidationContext(final ControllerService controllerService) { + return null; + } + + @Override + public ExpressionLanguageCompiler newExpressionLanguageCompiler() { + return null; + } + + @Override + public PropertyValue newPropertyValue(final String rawValue) { + return new StandardPropertyValue(rawValue, getControllerServiceLookup(), parameterLookup); + } + + @Override + public Map getProperties() { + final Map properties = new HashMap<>(); + for (final Map.Entry entry : rawValues.entrySet()) { + final PropertyDescriptor descriptor = new PropertyDescriptor.Builder() + .name(entry.getKey()) + .build(); + properties.put(descriptor, entry.getValue()); + } + + return properties; + } + + @Override + public String getAnnotationData() { + return null; + } + + @Override + public boolean isValidationRequired(final ControllerService controllerService) { + return false; + } + + @Override + public boolean isExpressionLanguagePresent(final String value) { + return false; + } + + @Override + public boolean isExpressionLanguageSupported(final String value) { + return false; + } + + @Override + public String getProcessGroupIdentifier() { + return null; + } + + @Override + public Collection getReferencedParameters(final String propertyName) { + return List.of(); + } + + @Override + public boolean isParameterDefined(final String parameterName) { + return parameterLookup.getParameter(parameterName).isPresent(); + } + + @Override + public boolean isParameterSet(final String parameterName) { + final Optional optionalParameter = parameterLookup.getParameter(parameterName); + if (optionalParameter.isEmpty()) { + return false; + } + + return optionalParameter.get().getValue() != null; + } + + @Override + public PropertyValue getProperty(final PropertyDescriptor propertyDescriptor) { + final String rawValue = rawValues.get(propertyDescriptor.getName()); + final String effectiveValue = rawValue == null ? propertyDescriptor.getDefaultValue() : rawValue; + return newPropertyValue(effectiveValue); + } + + @Override + public Map getAllProperties() { + return Collections.unmodifiableMap(rawValues); + } + + @Override + public String evaluateParameters(final String value) { + final ParameterParser parameterParser = new ExpressionLanguageAgnosticParameterParser(); + final ParameterTokenList parameterTokenList = parameterParser.parseTokens(value); + return parameterTokenList.substitute(parameterLookup); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/DescribedValueProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/DescribedValueProvider.java new file mode 100644 index 000000000000..7b5004fac7d2 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/DescribedValueProvider.java @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.DescribedValue; + +import java.util.List; + +public interface DescribedValueProvider { + + List fetchAllowableValues(String stepName, String propertyName); + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/GhostConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/GhostConnector.java new file mode 100644 index 000000000000..0508821e9d1f --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/GhostConnector.java @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +public class GhostConnector implements Connector { + private final String identifier; + private final String canonicalClassName; + private final List validationResults; + private final List configVerificationResults; + private final Exception causeOfGhost; + + public GhostConnector(final String identifier, final String canonicalClassName, final Exception causeOfGhost) { + this.identifier = identifier; + this.canonicalClassName = canonicalClassName; + this.causeOfGhost = causeOfGhost; + + validationResults = List.of(new ValidationResult.Builder() + .subject("Missing Connector") + .valid(false) + .explanation("Could not create Connector of type " + canonicalClassName) + .build()); + + configVerificationResults = List.of(new ConfigVerificationResult.Builder() + .verificationStepName("Create Connector") + .outcome(Outcome.FAILED) + .explanation("Could not create Connector of type " + canonicalClassName) + .build()); + } + + public Exception getCauseOfGhost() { + return causeOfGhost; + } + + @Override + public void initialize(final ConnectorInitializationContext initContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void start(final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public void stop(final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List validate(final FlowContext flowContext, final ConnectorValidationContext connectorValidationContext) { + return validationResults; + } + + @Override + public List validateConfigurationStep(final ConfigurationStep configurationStep, final ConnectorConfigurationContext connectorConfigurationContext, + final ConnectorValidationContext connectorValidationContext) { + return validationResults; + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void onConfigurationStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public void abortUpdate(final FlowContext workingContext, final Throwable throwable) { + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext workingContext) { + return configVerificationResults; + } + + @Override + public List verify(final FlowContext flowContext) { + return configVerificationResults; + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext, final String filter) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext) { + return List.of(); + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + return CompletableFuture.completedFuture(null); + } + + @Override + public String toString() { + return "GhostConnector[id=" + identifier + ", type=" + canonicalClassName + "]"; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandaloneConnectorRequestReplicator.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandaloneConnectorRequestReplicator.java new file mode 100644 index 000000000000..5880530745c6 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandaloneConnectorRequestReplicator.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.controller.flow.FlowManager; + +public class StandaloneConnectorRequestReplicator implements ConnectorRequestReplicator { + + private FlowManager flowManager; + + @Override + public ConnectorState getState(final String connectorId) { + final ConnectorNode connectorNode = flowManager.getConnector(connectorId); + if (connectorNode == null) { + throw new IllegalArgumentException("No connector found with id: " + connectorId); + } + + return connectorNode.getCurrentState(); + } + + @Override + public void setFlowManager(final FlowManager flowManager) { + this.flowManager = flowManager; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardComponentBundleLookup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardComponentBundleLookup.java new file mode 100644 index 000000000000..cc91698827f0 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardComponentBundleLookup.java @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.nar.ExtensionManager; + +import java.util.List; +import java.util.Optional; + +public class StandardComponentBundleLookup implements ComponentBundleLookup { + private final ExtensionManager extensionManager; + + public StandardComponentBundleLookup(final ExtensionManager extensionManager) { + this.extensionManager = extensionManager; + } + + @Override + public List getAvailableBundles(final String componentType) { + final List bundles = extensionManager.getBundles(componentType); + return bundles.stream() + .map(this::convertBundle) + .toList(); + } + + @Override + public Optional getLatestBundle(final String componentType) { + final List availableBundles = getAvailableBundles(componentType); + if (availableBundles.isEmpty()) { + return Optional.empty(); + } + + Bundle newest = null; + for (final Bundle bundle : availableBundles) { + if (newest == null || compareVersion(bundle.getVersion(), newest.getVersion()) > 0) { + newest = bundle; + } + } + + return Optional.ofNullable(newest); + } + + private int compareVersion(final String v1, final String v2) { + final String baseVersion1 = getBaseVersion(v1); + final String baseVersion2 = getBaseVersion(v2); + + final String[] parts1 = baseVersion1.split("\\."); + final String[] parts2 = baseVersion2.split("\\."); + + final int length = Math.max(parts1.length, parts2.length); + for (int i = 0; i < length; i++) { + final String part1Str = i < parts1.length ? parts1[i] : "0"; + final String part2Str = i < parts2.length ? parts2[i] : "0"; + + final int comparison = compareVersionPart(part1Str, part2Str); + if (comparison != 0) { + return comparison; + } + } + + // Base versions are equal; compare qualifiers + final String qualifier1 = getQualifier(v1); + final String qualifier2 = getQualifier(v2); + return compareQualifiers(qualifier1, qualifier2); + } + + private int compareQualifiers(final String qualifier1, final String qualifier2) { + final int rank1 = getQualifierRank(qualifier1); + final int rank2 = getQualifierRank(qualifier2); + + if (rank1 != rank2) { + return Integer.compare(rank1, rank2); + } + + // Same qualifier type; compare numeric suffixes (e.g., RC2 > RC1, M4 > M3) + final int num1 = getQualifierNumber(qualifier1); + final int num2 = getQualifierNumber(qualifier2); + return Integer.compare(num1, num2); + } + + private int getQualifierRank(final String qualifier) { + if (qualifier == null || qualifier.isEmpty()) { + return 4; + } else if (qualifier.startsWith("RC")) { + return 3; + } else if (qualifier.startsWith("M")) { + return 2; + } else if (qualifier.equals("SNAPSHOT")) { + return 0; + } else { + return 1; + } + } + + private int getQualifierNumber(final String qualifier) { + if (qualifier == null || qualifier.isEmpty()) { + return 0; + } + + final StringBuilder digits = new StringBuilder(); + for (int i = 0; i < qualifier.length(); i++) { + final char c = qualifier.charAt(i); + if (Character.isDigit(c)) { + digits.append(c); + } + } + + if (digits.isEmpty()) { + return 0; + } + + try { + return Integer.parseInt(digits.toString()); + } catch (final NumberFormatException e) { + return 0; + } + } + + private String getQualifier(final String version) { + final int qualifierIndex = version.indexOf('-'); + return qualifierIndex > 0 ? version.substring(qualifierIndex + 1) : null; + } + + private int compareVersionPart(final String part1, final String part2) { + final Integer num1 = parseVersionPart(part1); + final Integer num2 = parseVersionPart(part2); + + if (num1 != null && num2 != null) { + return Integer.compare(num1, num2); + } else if (num1 != null) { + return 1; + } else if (num2 != null) { + return -1; + } else { + return part1.compareTo(part2); + } + } + + private Integer parseVersionPart(final String part) { + try { + return Integer.parseInt(part); + } catch (final NumberFormatException e) { + return null; + } + } + + private String getBaseVersion(final String version) { + final int qualifierIndex = version.indexOf('-'); + return qualifierIndex > 0 ? version.substring(0, qualifierIndex) : version; + } + + private Bundle convertBundle(final org.apache.nifi.bundle.Bundle bundle) { + final BundleCoordinate coordinate = bundle.getBundleDetails().getCoordinate(); + return new Bundle(coordinate.getGroup(), coordinate.getId(), coordinate.getVersion()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorAction.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorAction.java new file mode 100644 index 000000000000..3989bbe02c3e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorAction.java @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.util.Objects; + +public class StandardConnectorAction implements ConnectorAction { + + private final String name; + private final String description; + private final boolean allowed; + private final String reasonNotAllowed; + + public StandardConnectorAction(final String name, final String description, final boolean allowed, final String reasonNotAllowed) { + this.name = Objects.requireNonNull(name, "name is required"); + this.description = Objects.requireNonNull(description, "description is required"); + this.allowed = allowed; + this.reasonNotAllowed = reasonNotAllowed; + } + + @Override + public String getName() { + return name; + } + + @Override + public String getDescription() { + return description; + } + + @Override + public boolean isAllowed() { + return allowed; + } + + @Override + public String getReasonNotAllowed() { + return reasonNotAllowed; + } + + @Override + public boolean equals(final Object other) { + if (other == null || getClass() != other.getClass()) { + return false; + } + final StandardConnectorAction that = (StandardConnectorAction) other; + return allowed == that.allowed + && Objects.equals(name, that.name) + && Objects.equals(description, that.description) + && Objects.equals(reasonNotAllowed, that.reasonNotAllowed); + } + + @Override + public int hashCode() { + return Objects.hash(name, description, allowed, reasonNotAllowed); + } + + @Override + public String toString() { + if (reasonNotAllowed == null) { + return "StandardConnectorAction[name=" + name + ", allowed=" + allowed + "]"; + } + return "StandardConnectorAction[name=" + name + ", allowed=" + allowed + ", reason=" + reasonNotAllowed + "]"; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorConfigurationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorConfigurationContext.java new file mode 100644 index 000000000000..c37a246933ea --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorConfigurationContext.java @@ -0,0 +1,323 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretProvider; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; + +public class StandardConnectorConfigurationContext implements MutableConnectorConfigurationContext, Cloneable { + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorConfigurationContext.class); + + private final ReadWriteLock rwLock = new ReentrantReadWriteLock(); + private final Lock readLock = rwLock.readLock(); + private final Lock writeLock = rwLock.writeLock(); + private final AssetManager assetManager; + private final SecretsManager secretsManager; + + final Map propertyConfigurations = new HashMap<>(); + final Map resolvedPropertyConfigurations = new HashMap<>(); + + public StandardConnectorConfigurationContext(final AssetManager assetManager, final SecretsManager secretsManager) { + this.assetManager = assetManager; + this.secretsManager = secretsManager; + } + + @Override + public ConnectorPropertyValue getProperty(final String stepName, final String propertyName) { + return getProperty(stepName, propertyName, null); + } + + private ConnectorPropertyValue getProperty(final String stepName, final String propertyName, final String defaultValue) { + readLock.lock(); + try { + final StepConfiguration resolvedConfig = resolvedPropertyConfigurations.get(stepName); + if (resolvedConfig == null) { + return new StandardConnectorPropertyValue(defaultValue); + } + + final ConnectorValueReference valueReference = resolvedConfig.getPropertyValue(propertyName); + if (valueReference != null) { + // The resolvedPropertyConfigurations contains only StringLiteralValue references. + return new StandardConnectorPropertyValue(((StringLiteralValue) valueReference).getValue()); + } + + // Property not found + return new StandardConnectorPropertyValue(defaultValue); + } finally { + readLock.unlock(); + } + } + + @Override + public ConnectorPropertyValue getProperty(final ConfigurationStep configurationStep, final ConnectorPropertyDescriptor connectorPropertyDescriptor) { + return getProperty(configurationStep.getName(), connectorPropertyDescriptor.getName(), connectorPropertyDescriptor.getDefaultValue()); + } + + @Override + public Set getPropertyNames(final String configurationStepName) { + readLock.lock(); + try { + final StepConfiguration config = propertyConfigurations.get(configurationStepName); + if (config == null) { + return Collections.emptySet(); + } + + return new HashSet<>(config.getPropertyValues().keySet()); + } finally { + readLock.unlock(); + } + } + + @Override + public Set getPropertyNames(final ConfigurationStep configurationStep) { + return getPropertyNames(configurationStep.getName()); + } + + @Override + public StepConfigurationContext scopedToStep(final String stepName) { + return new StandardStepConfigurationContext(stepName, this); + } + + @Override + public StepConfigurationContext scopedToStep(final ConfigurationStep configurationStep) { + return scopedToStep(configurationStep.getName()); + } + + @Override + public StandardConnectorConfigurationContext createWithOverrides(final String stepName, final Map propertyOverrides) { + final StandardConnectorConfigurationContext created = new StandardConnectorConfigurationContext(assetManager, secretsManager); + readLock.lock(); + try { + for (final Map.Entry stepEntry : propertyConfigurations.entrySet()) { + final String existingStepName = stepEntry.getKey(); + final StepConfiguration existingConfig = stepEntry.getValue(); + + if (!existingStepName.equals(stepName)) { + created.setProperties(existingStepName, new StepConfiguration(new HashMap<>(existingConfig.getPropertyValues()))); + continue; + } + + final Map mergedProperties = new HashMap<>(existingConfig.getPropertyValues()); + for (final Map.Entry override : propertyOverrides.entrySet()) { + final String propertyValue = override.getValue(); + if (propertyValue == null) { + mergedProperties.remove(override.getKey()); + continue; + } + + mergedProperties.put(override.getKey(), new StringLiteralValue(propertyValue)); + } + created.setProperties(stepName, new StepConfiguration(mergedProperties)); + } + + return created; + } finally { + readLock.unlock(); + } + } + + + @Override + public ConfigurationUpdateResult setProperties(final String stepName, final StepConfiguration configuration) { + writeLock.lock(); + try { + final StepConfiguration existingConfig = propertyConfigurations.get(stepName); + final Map existingProperties = existingConfig != null ? existingConfig.getPropertyValues() : new HashMap<>(); + final Map mergedProperties = new HashMap<>(existingProperties); + mergedProperties.putAll(configuration.getPropertyValues()); + + final StepConfiguration resolvedConfig = resolvePropertyValues(mergedProperties); + + final StepConfiguration updatedStepConfig = new StepConfiguration(new HashMap<>(mergedProperties)); + final StepConfiguration existingStepConfig = this.propertyConfigurations.put(stepName, updatedStepConfig); + final StepConfiguration existingResolvedStepConfig = this.resolvedPropertyConfigurations.put(stepName, resolvedConfig); + + if (Objects.equals(existingStepConfig, updatedStepConfig) && Objects.equals(existingResolvedStepConfig, resolvedConfig)) { + return ConfigurationUpdateResult.NO_CHANGES; + } + + return ConfigurationUpdateResult.CHANGES_MADE; + } finally { + writeLock.unlock(); + } + } + + private StepConfiguration resolvePropertyValues(final Map propertyValues) { + final Map resolvedProperties = new HashMap<>(); + for (final Map.Entry entry : propertyValues.entrySet()) { + final ConnectorValueReference resolved = resolve(entry.getValue()); + resolvedProperties.put(entry.getKey(), resolved); + } + return new StepConfiguration(resolvedProperties); + } + + private ConnectorValueReference resolve(final ConnectorValueReference reference) { + if (reference == null) { + return null; + } + + try { + return switch (reference) { + case StringLiteralValue stringLiteral -> stringLiteral; + case AssetReference assetReference -> resolveAssetReferences(assetReference); + case SecretReference secretReference -> new StringLiteralValue(getSecretValue(secretReference)); + }; + } catch (final IOException ioe) { + throw new UncheckedIOException("Unable to obtain Secrets from Secret Manager", ioe); + } + } + + private StringLiteralValue resolveAssetReferences(final AssetReference assetReference) { + final Set resolvedAssetValues = new HashSet<>(); + for (final String assetId : assetReference.getAssetIdentifiers()) { + assetManager.getAsset(assetId) + .map(Asset::getFile) + .map(File::getAbsolutePath) + .ifPresent(resolvedAssetValues::add); + } + + logger.debug("Resolved {} to {}", assetReference, resolvedAssetValues); + return new StringLiteralValue(String.join(",", resolvedAssetValues)); + } + + private String getSecretValue(final SecretReference secretReference) throws IOException { + final SecretProvider provider = getSecretProvider(secretReference); + if (provider == null) { + return null; + } + + final List secrets = provider.getSecrets(List.of(secretReference.getFullyQualifiedName())); + return secrets.isEmpty() ? null : secrets.getFirst().getValue(); + } + + private SecretProvider getSecretProvider(final SecretReference secretReference) { + final Set providers = secretsManager.getSecretProviders(); + for (final SecretProvider provider : providers) { + if (Objects.equals(provider.getProviderId(), secretReference.getProviderId())) { + return provider; + } + } + + for (final SecretProvider provider : providers) { + if (Objects.equals(provider.getProviderName(), secretReference.getProviderName())) { + return provider; + } + } + + // Try to find by Fully Qualified Name prefix + final String fqn = secretReference.getFullyQualifiedName(); + if (fqn != null) { + for (final SecretProvider provider : providers) { + if (fqn.startsWith(provider.getProviderName() + ".")) { + return provider; + } + } + } + + return null; + } + + @Override + public ConfigurationUpdateResult replaceProperties(final String stepName, final StepConfiguration configuration) { + writeLock.lock(); + try { + final StepConfiguration resolvedConfig = resolvePropertyValues(configuration.getPropertyValues()); + + final StepConfiguration updatedStepConfig = new StepConfiguration(new HashMap<>(configuration.getPropertyValues())); + final StepConfiguration existingStepConfig = this.propertyConfigurations.put(stepName, updatedStepConfig); + final StepConfiguration existingResolvedStepConfig = this.resolvedPropertyConfigurations.put(stepName, resolvedConfig); + + if (Objects.equals(existingStepConfig, updatedStepConfig) && Objects.equals(existingResolvedStepConfig, resolvedConfig)) { + return ConfigurationUpdateResult.NO_CHANGES; + } + + return ConfigurationUpdateResult.CHANGES_MADE; + } finally { + writeLock.unlock(); + } + } + + @Override + public void resolvePropertyValues() { + writeLock.lock(); + try { + for (final Map.Entry entry : propertyConfigurations.entrySet()) { + final String stepName = entry.getKey(); + final StepConfiguration stepConfig = entry.getValue(); + final Map stepProperties = stepConfig.getPropertyValues(); + + final StepConfiguration resolvedConfig = resolvePropertyValues(stepProperties); + this.resolvedPropertyConfigurations.put(stepName, resolvedConfig); + } + } finally { + writeLock.unlock(); + } + } + + @Override + public ConnectorConfiguration toConnectorConfiguration() { + readLock.lock(); + try { + final Set stepConfigs = new HashSet<>(); + for (final Map.Entry entry : propertyConfigurations.entrySet()) { + final String stepName = entry.getKey(); + final StepConfiguration config = entry.getValue(); + final StepConfiguration configCopy = new StepConfiguration(new HashMap<>(config.getPropertyValues())); + + stepConfigs.add(new NamedStepConfiguration(stepName, configCopy)); + } + + return new ConnectorConfiguration(stepConfigs); + } finally { + readLock.unlock(); + } + } + + @Override + public MutableConnectorConfigurationContext clone() { + readLock.lock(); + try { + final StandardConnectorConfigurationContext cloned = new StandardConnectorConfigurationContext(assetManager, secretsManager); + for (final Map.Entry entry : this.propertyConfigurations.entrySet()) { + cloned.setProperties(entry.getKey(), new StepConfiguration(new HashMap<>(entry.getValue().getPropertyValues()))); + } + return cloned; + } finally { + readLock.unlock(); + } + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorConfigurationProviderInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorConfigurationProviderInitializationContext.java new file mode 100644 index 000000000000..99f3bee1207e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorConfigurationProviderInitializationContext.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; + +import java.util.Map; +import java.util.Objects; + +public class StandardConnectorConfigurationProviderInitializationContext implements ConnectorConfigurationProviderInitializationContext { + + private final Map properties; + private final AssetManager assetManager; + + public StandardConnectorConfigurationProviderInitializationContext(final Map properties, final AssetManager assetManager) { + this.properties = Map.copyOf(Objects.requireNonNull(properties, "Properties is required")); + this.assetManager = Objects.requireNonNull(assetManager, "AssetManager is required"); + } + + @Override + public Map getProperties() { + return properties; + } + + @Override + public AssetManager getAssetManager() { + return assetManager; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorInitializationContext.java new file mode 100644 index 000000000000..002f2a8aa2f8 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorInitializationContext.java @@ -0,0 +1,205 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.logging.ComponentLog; + +import java.util.List; +import java.util.function.Consumer; + +public class StandardConnectorInitializationContext implements FrameworkConnectorInitializationContext { + private final String identifier; + private final String name; + private final ComponentLog componentLog; + private final SecretsManager secretsManager; + private final AssetManager assetManager; + private final ComponentBundleLookup componentBundleLookup; + + + protected StandardConnectorInitializationContext(final Builder builder) { + this.identifier = builder.identifier; + this.name = builder.name; + this.componentLog = builder.componentLog; + this.secretsManager = builder.secretsManager; + this.assetManager = builder.assetManager; + this.componentBundleLookup = builder.componentBundleLookup; + } + + @Override + public String getIdentifier() { + return identifier; + } + + @Override + public String getName() { + return name; + } + + @Override + public ComponentLog getLogger() { + return componentLog; + } + + @Override + public ComponentBundleLookup getComponentBundleLookup() { + return componentBundleLookup; + } + + @Override + public SecretsManager getSecretsManager() { + return secretsManager; + } + + @Override + public AssetManager getAssetManager() { + return assetManager; + } + + @Override + public void updateFlow(final FlowContext flowContext, final VersionedExternalFlow versionedExternalFlow, + final BundleCompatibility bundleCompatability) throws FlowUpdateException { + if (!(flowContext instanceof final FrameworkFlowContext frameworkFlowContext)) { + throw new IllegalArgumentException("FlowContext is not an instance provided by the framework"); + } + + resolveBundles(versionedExternalFlow.getFlowContents(), bundleCompatability); + frameworkFlowContext.updateFlow(versionedExternalFlow, assetManager); + } + + protected void resolveBundles(final VersionedProcessGroup group, final BundleCompatibility bundleCompatability) { + if (bundleCompatability == BundleCompatibility.REQUIRE_EXACT_BUNDLE) { + return; + } + + if (group.getProcessors() != null) { + for (final VersionedProcessor processor : group.getProcessors()) { + resolveBundle(processor.getType(), processor.getBundle(), bundleCompatability, processor::setBundle); + } + } + + if (group.getControllerServices() != null) { + for (final VersionedControllerService service : group.getControllerServices()) { + resolveBundle(service.getType(), service.getBundle(), bundleCompatability, service::setBundle); + } + } + + if (group.getProcessGroups() != null) { + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + resolveBundles(childGroup, bundleCompatability); + } + } + } + + private void resolveBundle(final String componentType, final Bundle currentBundle, + final BundleCompatibility bundleCompatability, final Consumer bundleSetter) { + final List availableBundles = componentBundleLookup.getAvailableBundles(componentType); + + if (availableBundles.contains(currentBundle)) { + return; + } + + if (availableBundles.isEmpty()) { + return; + } + + switch (bundleCompatability) { + case RESOLVE_BUNDLE: + if (availableBundles.size() == 1) { + final Bundle resolvedBundle = availableBundles.getFirst(); + componentLog.debug("Resolved bundle for {} from {} to {}", componentType, currentBundle, resolvedBundle); + bundleSetter.accept(resolvedBundle); + } + break; + case RESOLVE_NEWEST_BUNDLE: + componentBundleLookup.getLatestBundle(componentType).ifPresent(latestBundle -> { + componentLog.debug("Resolved bundle for {} from {} to {}", componentType, currentBundle, latestBundle); + bundleSetter.accept(latestBundle); + }); + break; + default: + break; + } + } + + private void updateParameterContext(final VersionedProcessGroup group, final String parameterContextName) { + group.setParameterContextName(parameterContextName); + if (group.getProcessGroups() != null) { + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + updateParameterContext(childGroup, parameterContextName); + } + } + } + + public static class Builder implements FrameworkConnectorInitializationContextBuilder { + private String identifier; + private String name; + private ComponentLog componentLog; + private SecretsManager secretsManager; + private AssetManager assetManager; + private ComponentBundleLookup componentBundleLookup; + + @Override + public Builder identifier(final String identifier) { + this.identifier = identifier; + return this; + } + + @Override + public Builder name(final String name) { + this.name = name; + return this; + } + + @Override + public Builder componentLog(final ComponentLog componentLog) { + this.componentLog = componentLog; + return this; + } + + @Override + public Builder secretsManager(final SecretsManager secretsManager) { + this.secretsManager = secretsManager; + return this; + } + + @Override + public Builder assetManager(final AssetManager assetManager) { + this.assetManager = assetManager; + return this; + } + + @Override + public Builder componentBundleLookup(final ComponentBundleLookup bundleLookup) { + this.componentBundleLookup = bundleLookup; + return this; + } + + @Override + public StandardConnectorInitializationContext build() { + return new StandardConnectorInitializationContext(this); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java new file mode 100644 index 000000000000..d2f53dcdd43a --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java @@ -0,0 +1,1605 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.authorization.Resource; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.resource.ResourceFactory; +import org.apache.nifi.authorization.resource.ResourceType; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.validation.DisabledServiceValidationResult; +import org.apache.nifi.components.validation.ValidationState; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.components.validation.ValidationTrigger; +import org.apache.nifi.connectable.FlowFileActivity; +import org.apache.nifi.connectable.FlowFileTransferCounts; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.controller.queue.DropFlowFileStatus; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.engine.FlowEngine; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedConfigurationStep; +import org.apache.nifi.flow.VersionedConnectorValueReference; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.nar.NarCloseable; +import org.apache.nifi.util.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.OptionalLong; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.Future; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; +import java.util.stream.Collectors; + +public class StandardConnectorNode implements ConnectorNode { + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorNode.class); + + private final String identifier; + private final FlowManager flowManager; + private final ExtensionManager extensionManager; + private final Authorizable parentAuthorizable; + private final ConnectorDetails connectorDetails; + private final String componentType; + private final String componentCanonicalClass; + private final BundleCoordinate bundleCoordinate; + private final ConnectorStateTransition stateTransition; + private final AtomicReference versionedComponentId = new AtomicReference<>(); + private final FlowContextFactory flowContextFactory; + private final FrameworkFlowContext activeFlowContext; + + private final AtomicReference validationState = new AtomicReference<>(new ValidationState(ValidationStatus.VALIDATING, Collections.emptyList())); + private final ConnectorValidationTrigger validationTrigger; + private final boolean extensionMissing; + private volatile boolean triggerValidation = true; + private final AtomicReference> drainFutureRef = new AtomicReference<>(); + private volatile ValidationResult unresolvedBundleValidationResult = null; + + private volatile FrameworkFlowContext workingFlowContext; + + private volatile String name; + private volatile FrameworkConnectorInitializationContext initializationContext; + + + public StandardConnectorNode(final String identifier, final FlowManager flowManager, final ExtensionManager extensionManager, + final Authorizable parentAuthorizable, final ConnectorDetails connectorDetails, final String componentType, final String componentCanonicalClass, + final MutableConnectorConfigurationContext configurationContext, + final ConnectorStateTransition stateTransition, final FlowContextFactory flowContextFactory, + final ConnectorValidationTrigger validationTrigger, final boolean extensionMissing) { + + this.identifier = identifier; + this.flowManager = flowManager; + this.extensionManager = extensionManager; + this.parentAuthorizable = parentAuthorizable; + this.connectorDetails = connectorDetails; + this.componentType = componentType; + this.componentCanonicalClass = componentCanonicalClass; + this.bundleCoordinate = connectorDetails.getBundleCoordinate(); + this.stateTransition = stateTransition; + this.flowContextFactory = flowContextFactory; + this.validationTrigger = validationTrigger; + this.extensionMissing = extensionMissing; + + this.name = connectorDetails.getConnector().getClass().getSimpleName(); + + final Bundle activeFlowBundle = new Bundle(bundleCoordinate.getGroup(), bundleCoordinate.getId(), bundleCoordinate.getVersion()); + this.activeFlowContext = flowContextFactory.createActiveFlowContext(identifier, connectorDetails.getComponentLog(), activeFlowBundle); + } + + @Override + public String getName() { + return name; + } + + @Override + public void setName(final String name) { + this.name = name; + } + + @Override + public void transitionStateForUpdating() { + final ConnectorState initialState = getCurrentState(); + if (initialState == ConnectorState.UPDATING || initialState == ConnectorState.PREPARING_FOR_UPDATE) { + return; + } + + stateTransition.setDesiredState(ConnectorState.UPDATING); + stateTransition.setCurrentState(ConnectorState.PREPARING_FOR_UPDATE); + } + + @Override + public void prepareForUpdate() throws FlowUpdateException { + if (getCurrentState() != ConnectorState.PREPARING_FOR_UPDATE) { + throw new IllegalStateException("Cannot prepare update for " + this + " because its state is currently " + getCurrentState() + + "; it must be PREPARING_FOR_UPDATE."); + } + if (getDesiredState() != ConnectorState.UPDATING) { + throw new IllegalStateException("Cannot prepare update for " + this + " because its desired state is currently " + getDesiredState() + + "; it must be UPDATING."); + } + + logger.debug("Preparing {} for update", this); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + getConnector().prepareForUpdate(workingFlowContext, activeFlowContext); + stateTransition.setCurrentState(ConnectorState.UPDATING); + logger.debug("Successfully prepared {} for update", this); + } catch (final Throwable t) { + logger.error("Failed to prepare update for {}", this, t); + + try { + abortUpdate(t); + } catch (final Throwable abortFailure) { + logger.error("Failed to abort update preparation for {}", this, abortFailure); + } + + throw t; + } + } + + @Override + public void inheritConfiguration(final List activeConfig, final List workingConfig, + final Bundle flowContextBundle) throws FlowUpdateException { + + logger.debug("Inheriting configuration for {}", this); + final MutableConnectorConfigurationContext configurationContext = createConfigurationContext(activeConfig); + final FrameworkFlowContext inheritContext = flowContextFactory.createWorkingFlowContext(identifier, + connectorDetails.getComponentLog(), configurationContext, flowContextBundle); + + // Apply the update for the active config + applyUpdate(inheritContext); + + // Configure the working config but do not apply + for (final VersionedConfigurationStep step : workingConfig) { + final StepConfiguration stepConfig = createStepConfiguration(step); + setConfiguration(step.getName(), stepConfig, true); + } + + logger.debug("Successfully inherited configuration for {}", this); + } + + private StepConfiguration createStepConfiguration(final VersionedConfigurationStep step) { + final Map convertedProperties = new HashMap<>(); + if (step.getProperties() != null) { + for (final Map.Entry entry : step.getProperties().entrySet()) { + final ConnectorValueReference valueReference = createValueReference(entry.getValue()); + convertedProperties.put(entry.getKey(), valueReference); + } + } + + return new StepConfiguration(convertedProperties); + } + + private MutableConnectorConfigurationContext createConfigurationContext(final List flowConfiguration) { + final StandardConnectorConfigurationContext configurationContext = new StandardConnectorConfigurationContext( + initializationContext.getAssetManager(), initializationContext.getSecretsManager()); + + for (final VersionedConfigurationStep versionedConfigStep : flowConfiguration) { + final StepConfiguration stepConfig = createStepConfiguration(versionedConfigStep); + configurationContext.setProperties(versionedConfigStep.getName(), stepConfig); + } + + return configurationContext; + } + + private ConnectorValueReference createValueReference(final VersionedConnectorValueReference versionedReference) { + final ConnectorValueType valueType = ConnectorValueType.valueOf(versionedReference.getValueType()); + return switch (valueType) { + case STRING_LITERAL -> new StringLiteralValue(versionedReference.getValue()); + case ASSET_REFERENCE -> new AssetReference(versionedReference.getAssetIds()); + case SECRET_REFERENCE -> new SecretReference(versionedReference.getProviderId(), versionedReference.getProviderName(), + versionedReference.getSecretName(), versionedReference.getFullyQualifiedSecretName()); + }; + } + + @Override + public void applyUpdate() throws FlowUpdateException { + try { + applyUpdate(workingFlowContext); + } catch (final FlowUpdateException e) { + // Since we failed to update, make sure that we stop the Connector. Note that we do not do this for all + // throwables because IllegalStateException for example indicates that we did not even attempt to perform the update. + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connectorDetails.getConnector().getClass(), getIdentifier())) { + connectorDetails.getConnector().stop(activeFlowContext); + } catch (final Throwable stopThrowable) { + e.addSuppressed(stopThrowable); + } + + throw e; + } + } + + private void applyUpdate(final FrameworkFlowContext contextToInherit) throws FlowUpdateException { + final ConnectorState currentState = getCurrentState(); + if (currentState != ConnectorState.UPDATING) { + throw new IllegalStateException("Cannot finish update for " + this + " because its state is currently " + currentState + + "; it must be UPDATING."); + } + + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + getConnector().applyUpdate(contextToInherit, activeFlowContext); + + // Update the active flow context based on the properties of the provided context, as the connector has now been updated. + final ConnectorConfiguration workingConfig = contextToInherit.getConfigurationContext().toConnectorConfiguration(); + for (final NamedStepConfiguration stepConfig : workingConfig.getNamedStepConfigurations()) { + activeFlowContext.getConfigurationContext().replaceProperties(stepConfig.stepName(), stepConfig.configuration()); + } + + getComponentLog().info("Working Context has been applied to Active Context"); + + // The update has been completed. Tear down and recreate the working flow context to ensure it is in a clean state. + resetValidationState(); + recreateWorkingFlowContext(); + } catch (final Throwable t) { + logger.error("Failed to finish update for {}", this, t); + stateTransition.setCurrentState(ConnectorState.UPDATE_FAILED); + stateTransition.setDesiredState(ConnectorState.UPDATE_FAILED); + + throw new FlowUpdateException("Failed to finish update for " + this, t); + } + + stateTransition.setCurrentState(ConnectorState.UPDATED); + stateTransition.setDesiredState(ConnectorState.UPDATED); + logger.info("Successfully applied update for {}", this); + } + + private void destroyWorkingContext() { + if (this.workingFlowContext == null) { + return; + } + + try { + workingFlowContext.getManagedProcessGroup().purge().get(1, TimeUnit.MINUTES); + } catch (final Exception e) { + logger.warn("Failed to purge working flow context for {}", this, e); + } + + flowManager.onProcessGroupRemoved(workingFlowContext.getManagedProcessGroup()); + + this.workingFlowContext = null; + } + + @Override + public void abortUpdate(final Throwable cause) { + stateTransition.setCurrentState(ConnectorState.UPDATE_FAILED); + stateTransition.setDesiredState(ConnectorState.UPDATE_FAILED); + + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + getConnector().abortUpdate(workingFlowContext, cause); + } + logger.debug("Aborted update for {}", this); + } + + @Override + public void markInvalid(final String subject, final String explanation) { + validationState.set(new ValidationState(ValidationStatus.INVALID, List.of(new ValidationResult.Builder() + .subject(subject) + .valid(false) + .explanation(explanation) + .build()))); + } + + @Override + public void setConfiguration(final String stepName, final StepConfiguration configuration) throws FlowUpdateException { + setConfiguration(stepName, configuration, false); + } + + private void setConfiguration(final String stepName, final StepConfiguration configuration, final boolean forceOnConfigurationStepConfigured) throws FlowUpdateException { + // Update properties and check if the configuration changed. + final ConfigurationUpdateResult updateResult = workingFlowContext.getConfigurationContext().setProperties(stepName, configuration); + if (updateResult == ConfigurationUpdateResult.NO_CHANGES && !forceOnConfigurationStepConfigured) { + return; + } + + // If there were changes, trigger Processor to be notified of the change. + final Connector connector = connectorDetails.getConnector(); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connector.getClass(), getIdentifier())) { + logger.debug("Notifying {} of configuration change for configuration step {}", this, stepName); + connector.onConfigurationStepConfigured(stepName, workingFlowContext); + logger.debug("Successfully set configuration for step {} on {}", stepName, this); + } catch (final FlowUpdateException e) { + throw e; + } catch (final Exception e) { + logger.error("Failed to invoke onConfigured for {}", this, e); + throw new RuntimeException("Failed to invoke onConfigured for " + this, e); + } + } + + @Override + public ConnectorState getCurrentState() { + return stateTransition.getCurrentState(); + } + + @Override + public ConnectorState getDesiredState() { + return stateTransition.getDesiredState(); + } + + + @Override + public Optional getIdleDuration() { + final ProcessGroup processGroup = getActiveFlowContext().getManagedProcessGroup(); + final FlowFileActivity activity = processGroup.getFlowFileActivity(); + final OptionalLong lastActivityTimestamp = activity.getLatestActivityTime(); + if (lastActivityTimestamp.isEmpty()) { + return Optional.empty(); + } + + if (processGroup.isDataQueued()) { + return Optional.empty(); + } + + final Duration idleDuration = Duration.ofMillis(System.currentTimeMillis() - lastActivityTimestamp.getAsLong()); + return Optional.of(idleDuration); + } + + @Override + public FlowFileTransferCounts getFlowFileTransferCounts() { + return getActiveFlowContext().getManagedProcessGroup().getFlowFileActivity().getTransferCounts(); + } + + @Override + public Future start(final FlowEngine scheduler) { + final CompletableFuture startCompleteFuture = new CompletableFuture<>(); + start(scheduler, startCompleteFuture); + return startCompleteFuture; + } + + private void start(final FlowEngine scheduler, final CompletableFuture startCompleteFuture) { + try { + stateTransition.setDesiredState(ConnectorState.RUNNING); + activeFlowContext.getConfigurationContext().resolvePropertyValues(); + + verifyCanStart(); + + final ConnectorState currentState = getCurrentState(); + switch (currentState) { + case STARTING -> { + logger.debug("{} is already starting; adding future to pending start futures", this); + stateTransition.addPendingStartFuture(startCompleteFuture); + } + case RUNNING -> { + logger.debug("{} is already {}; will not attempt to start", this, currentState); + startCompleteFuture.complete(null); + } + case STOPPING -> { + // We have set the Desired State to RUNNING so when the Connector fully stops, it will be started again automatically + logger.info("{} is currently stopping so will not trigger Connector to start until it has fully stopped", this); + stateTransition.addPendingStartFuture(startCompleteFuture); + } + case STOPPED, PREPARING_FOR_UPDATE, UPDATED -> { + stateTransition.setCurrentState(ConnectorState.STARTING); + scheduler.schedule(() -> startComponent(scheduler, startCompleteFuture), 0, TimeUnit.SECONDS); + } + default -> { + logger.warn("{} is in state {} and cannot be started", this, currentState); + stateTransition.addPendingStartFuture(startCompleteFuture); + } + } + } catch (final Exception e) { + logger.error("Failed to start {}", this, e); + startCompleteFuture.completeExceptionally(e); + } + } + + @Override + public Future stop(final FlowEngine scheduler) { + logger.info("Stopping {}", this); + final CompletableFuture stopCompleteFuture = new CompletableFuture<>(); + + stateTransition.setDesiredState(ConnectorState.STOPPED); + + boolean stateUpdated = false; + while (!stateUpdated) { + final ConnectorState currentState = getCurrentState(); + if (currentState == ConnectorState.STOPPED) { + logger.info("{} is already stopped.", this); + stopCompleteFuture.complete(null); + return stopCompleteFuture; + } + + if (currentState == ConnectorState.STOPPING) { + logger.debug("{} is already stopping; adding future to pending stop futures", this); + stateTransition.addPendingStopFuture(stopCompleteFuture); + return stopCompleteFuture; + } + + stateUpdated = stateTransition.trySetCurrentState(currentState, ConnectorState.STOPPING); + } + + scheduler.schedule(() -> stopComponent(scheduler, stopCompleteFuture), 0, TimeUnit.SECONDS); + + return stopCompleteFuture; + } + + @Override + public Future drainFlowFiles() { + logger.debug("Draining FlowFiles for {}", this); + requireStopped("drain FlowFiles", ConnectorState.DRAINING); + + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connectorDetails.getConnector().getClass(), getIdentifier())) { + getComponentLog().info("Draining FlowFiles from {}", this); + final CompletableFuture drainFuture = connectorDetails.getConnector().drainFlowFiles(activeFlowContext); + drainFutureRef.set(drainFuture); + + final CompletableFuture stateUpdateFuture = drainFuture.whenComplete((result, failureCause) -> { + drainFutureRef.set(null); + + if (failureCause == null) { + logger.info("Successfully drained FlowFiles for {}", this); + } else { + logger.error("Failed to drain FlowFiles for {}", this, failureCause); + } + + try { + connectorDetails.getConnector().stop(activeFlowContext); + } catch (final Exception e) { + logger.warn("Failed to stop {} after draining FlowFiles", this, e); + } + + stateTransition.setCurrentState(ConnectorState.STOPPED); + logger.info("All components of {} are now stopped after draining FlowFiles.", this); + }); + + return stateUpdateFuture; + } catch (final Throwable t) { + logger.error("Failed to drain FlowFiles for {}", this, t); + stateTransition.setCurrentState(ConnectorState.STOPPED); + throw t; + } + } + + @Override + public void cancelDrainFlowFiles() { + final Future future = this.drainFutureRef.getAndSet(null); + if (future == null) { + logger.debug("No active drain to cancel for {}; drain may have already completed", this); + return; + } + + future.cancel(true); + logger.info("Cancelled draining of FlowFiles for {}", this); + } + + @Override + public void verifyCancelDrainFlowFiles() throws IllegalStateException { + final ConnectorState state = getCurrentState(); + + // Allow if we're currently draining or if we're stopped; if stopped the cancel drain action will be a no-op + // but we don't want to throw an IllegalStateException in that case because doing so would mean that if one + // node in the cluster is stopped while another is draining we cannot cancel the drain. + if (state != ConnectorState.DRAINING && state != ConnectorState.STOPPED) { + throw new IllegalStateException("Cannot cancel draining of FlowFiles for " + this + " because its current state is " + state + "; it must be DRAINING."); + } + } + + @Override + public void verifyCanPurgeFlowFiles() throws IllegalStateException { + final ConnectorState desiredState = getDesiredState(); + if (desiredState != ConnectorState.STOPPED) { + throw new IllegalStateException("Cannot purge FlowFiles for " + this + " because its desired state is currently " + desiredState + "; it must be STOPPED."); + } + + final ConnectorState currentState = getCurrentState(); + if (currentState != ConnectorState.STOPPED) { + throw new IllegalStateException("Cannot purge FlowFiles for " + this + " because its current state is " + currentState + "; it must be STOPPED."); + } + } + + @Override + public Future purgeFlowFiles(final String requestor) { + logger.debug("Purging FlowFiles for {}", this); + requireStopped("purge FlowFiles", ConnectorState.PURGING); + + try { + final String dropRequestId = UUID.randomUUID().toString(); + final DropFlowFileStatus status = activeFlowContext.getManagedProcessGroup().dropAllFlowFiles(dropRequestId, requestor); + final CompletableFuture future = status.getCompletionFuture(); + final CompletableFuture stateUpdateFuture = future.whenComplete((result, failureCause) -> { + stateTransition.setCurrentState(ConnectorState.STOPPED); + + if (failureCause == null) { + logger.info("Successfully purged FlowFiles for {}", this); + } else { + logger.error("Failed to purge FlowFiles for {}", this, failureCause); + } + }); + return stateUpdateFuture; + } catch (final Throwable t) { + logger.error("Failed to purge FlowFiles for {}", this, t); + stateTransition.setCurrentState(ConnectorState.STOPPED); + throw t; + } + } + + private void requireStopped(final String action, final ConnectorState newState) { + final ConnectorState desiredState = getDesiredState(); + if (desiredState != ConnectorState.STOPPED) { + throw new IllegalStateException("Cannot " + action + " for " + this + " because its desired state is currently " + desiredState + "; it must be STOPPED."); + } + + boolean stateUpdated = false; + while (!stateUpdated) { + final ConnectorState currentState = getCurrentState(); + if (currentState != ConnectorState.STOPPED) { + throw new IllegalStateException("Cannot " + action + " for " + this + " because its current state is " + currentState + "; it must be STOPPED."); + } + + stateUpdated = stateTransition.trySetCurrentState(ConnectorState.STOPPED, newState); + } + } + + private void stopComponent(final FlowEngine scheduler, final CompletableFuture stopCompleteFuture) { + logger.debug("Stopping component for {}", this); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connectorDetails.getConnector().getClass(), getIdentifier())) { + connectorDetails.getConnector().stop(activeFlowContext); + } catch (final Exception e) { + logger.error("Failed to stop {}. Will try again in 10 seconds", this, e); + scheduler.schedule(() -> stopComponent(scheduler, stopCompleteFuture), 10, TimeUnit.SECONDS); + return; + } + + stateTransition.setCurrentState(ConnectorState.STOPPED); + stopCompleteFuture.complete(null); + logger.info("Successfully stopped {}", this); + + final ConnectorState desiredState = getDesiredState(); + if (desiredState == ConnectorState.RUNNING) { + logger.info("{} was requested to be RUNNING while it was stopping so will attempt to start again", this); + start(scheduler, new CompletableFuture<>()); + } + } + + private void startComponent(final ScheduledExecutorService scheduler, final CompletableFuture startCompleteFuture) { + logger.debug("Starting component for {}", this); + final ConnectorState desiredState = getDesiredState(); + if (desiredState != ConnectorState.RUNNING) { + logger.info("Will not start {} because the desired state is no longer RUNNING but is now {}", this, desiredState); + return; + } + + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connectorDetails.getConnector().getClass(), getIdentifier())) { + connectorDetails.getConnector().start(activeFlowContext); + } catch (final Exception e) { + logger.error("Failed to start {}. Will try again in 10 seconds", this, e); + scheduler.schedule(() -> startComponent(scheduler, startCompleteFuture), 10, TimeUnit.SECONDS); + return; + } + + stateTransition.setCurrentState(ConnectorState.RUNNING); + startCompleteFuture.complete(null); + logger.info("Successfully started {}", this); + } + + + @Override + public void verifyCanDelete() { + final QueueSize queueSize = getActiveFlowContext().getManagedProcessGroup().getQueueSize(); + if (queueSize.getObjectCount() > 0) { + throw new IllegalStateException("Cannot delete " + this + " because its Process Group has " + queueSize.getObjectCount() + + " FlowFiles queued; all FlowFiles must be removed before it can be deleted."); + } + + final ConnectorState currentState = getCurrentState(); + if (currentState == ConnectorState.STOPPED || currentState == ConnectorState.UPDATE_FAILED || currentState == ConnectorState.UPDATED) { + return; + } + + throw new IllegalStateException("Cannot delete " + this + " because its state is currently " + currentState + "; it must be stopped before it can be deleted."); + } + + @Override + public void verifyCanStart() { + final ValidationState state = performValidation(); + if (state.getStatus() != ValidationStatus.VALID) { + throw new IllegalStateException("Cannot start " + this + " because it is not valid: " + state.getValidationErrors()); + } + } + + @Override + public Connector getConnector() { + return connectorDetails.getConnector(); + } + + @Override + public String getComponentType() { + return componentType; + } + + @Override + public String getCanonicalClassName() { + return componentCanonicalClass; + } + + @Override + public BundleCoordinate getBundleCoordinate() { + return bundleCoordinate; + } + + @Override + public boolean isExtensionMissing() { + return extensionMissing; + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName) { + if (workingFlowContext == null) { + throw new IllegalStateException("Cannot fetch Allowable Values for %s.%s because %s is not being updated.".formatted( + stepName, propertyName, this)); + } + + workingFlowContext.getConfigurationContext().resolvePropertyValues(); + + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + return getConnector().fetchAllowableValues(stepName, propertyName, workingFlowContext); + } + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final String filter) { + if (workingFlowContext == null) { + throw new IllegalStateException("Cannot fetch Allowable Values for %s.%s because %s is not being updated.".formatted( + stepName, propertyName, this)); + } + + workingFlowContext.getConfigurationContext().resolvePropertyValues(); + + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + return getConnector().fetchAllowableValues(stepName, propertyName, workingFlowContext, filter); + } + } + + @Override + public void initializeConnector(final FrameworkConnectorInitializationContext initializationContext) { + logger.debug("Initializing {}", this); + this.initializationContext = initializationContext; + + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + getConnector().initialize(initializationContext); + } + + recreateWorkingFlowContext(); + logger.info("Successfully initialized {}", this); + } + + @Override + public void loadInitialFlow() throws FlowUpdateException { + logger.debug("Loading initial flow for {}", this); + if (initializationContext == null) { + throw new IllegalStateException("Cannot load initial flow because " + this + " has not been initialized yet."); + } + + final VersionedExternalFlow initialFlow; + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + initialFlow = getConnector().getInitialFlow(); + } + + if (initialFlow == null) { + logger.info("{} has no initial flow to load", this); + } else { + final ValidationResult unresolvedBundleResult = validateBundlesCanBeResolved(initialFlow.getFlowContents(), initializationContext.getComponentBundleLookup()); + + if (unresolvedBundleResult != null) { + logger.error("Cannot load initial flow for {} because some component bundles cannot be resolved: {}", this, unresolvedBundleResult.getExplanation()); + unresolvedBundleValidationResult = unresolvedBundleResult; + } else { + logger.info("Loading initial flow for {}", this); + // Update all RUNNING components to ENABLED before applying the initial flow so that components + // are not started before being configured. + stopComponents(initialFlow.getFlowContents()); + initializationContext.updateFlow(activeFlowContext, initialFlow, BundleCompatibility.RESOLVE_BUNDLE); + } + } + + resetValidationState(); + recreateWorkingFlowContext(); + } + + private void stopComponents(final VersionedProcessGroup group) { + for (final VersionedProcessor processor : group.getProcessors()) { + if (processor.getScheduledState() == ScheduledState.RUNNING) { + processor.setScheduledState(ScheduledState.ENABLED); + } + } + + for (final VersionedControllerService service : group.getControllerServices()) { + if (service.getScheduledState() == ScheduledState.RUNNING) { + service.setScheduledState(ScheduledState.ENABLED); + } + } + + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + stopComponents(childGroup); + } + } + + /** + * Ensures that all bundles required by the given Process Group can be resolved. We do this in order to make the Connector + * invalid if any Processor or Controller Service cannot be properly instantiated due to missing bundles. We intentionally + * differentiate between making the Connector invalid versus Ghosting the Connector for a few reasons: + *
    + *
  • + * Ghosting the Connector would prevent us from even getting the Configuration Steps, and it results in all Properties becoming sensitive. This can lead to confusion. + *
    • + *
  • + * The flow may change dynamically and so it's possible for a Connector to be valid given its initial flow and then become invalid + * based on configuration because the new configuration requires a new component that is unavailable. We would not suddenly change from + * a valid Connector to a Ghosted Connector, we could only become invalid. We do not want a missing component in the Initial Flow to be + * treated differently than a missing component from a subsequent flow update. + *
    • + *
  • + * Ghosting should be reserved for situations where the extension itself is missing. + *
    • + *
+ * + * @param group the process group to validate + * @param bundleLookup the bundle lookup + * @return a ValidationResult describing the missing bundles if any are missing; null if all bundles can be resolved + */ + private ValidationResult validateBundlesCanBeResolved(final VersionedProcessGroup group, final ComponentBundleLookup bundleLookup) { + final Set missingBundles = new HashSet<>(); + final Set missingProcessorTypes = new HashSet<>(); + final Set missingControllerServiceTypes = new HashSet<>(); + + collectUnresolvedBundles(group, bundleLookup, missingBundles, missingProcessorTypes, missingControllerServiceTypes); + + if (missingBundles.isEmpty()) { + return null; + } + + final StringBuilder explanation = new StringBuilder(); + explanation.append("%d Processors and %d Controller Services unavailable from %d missing bundles".formatted( + missingProcessorTypes.size(), missingControllerServiceTypes.size(), missingBundles.size())); + explanation.append("\nMissing Bundles: %s".formatted(missingBundles)); + if (!missingProcessorTypes.isEmpty()) { + explanation.append("\nMissing Processors: %s".formatted(missingProcessorTypes)); + } + if (!missingControllerServiceTypes.isEmpty()) { + explanation.append("\nMissing Controller Services: %s".formatted(missingControllerServiceTypes)); + } + + return new ValidationResult.Builder() + .subject("Missing Bundles") + .valid(false) + .explanation(explanation.toString()) + .build(); + } + + private void collectUnresolvedBundles(final VersionedProcessGroup group, final ComponentBundleLookup bundleLookup, + final Set missingBundles, final Set missingProcessorTypes, + final Set missingControllerServiceTypes) { + if (group.getProcessors() != null) { + for (final VersionedProcessor processor : group.getProcessors()) { + if (!isBundleResolvable(processor.getType(), processor.getBundle(), bundleLookup)) { + missingBundles.add(formatBundle(processor.getBundle())); + missingProcessorTypes.add(processor.getType()); + } + } + } + + if (group.getControllerServices() != null) { + for (final VersionedControllerService service : group.getControllerServices()) { + if (!isBundleResolvable(service.getType(), service.getBundle(), bundleLookup)) { + missingBundles.add(formatBundle(service.getBundle())); + missingControllerServiceTypes.add(service.getType()); + } + } + } + + if (group.getProcessGroups() != null) { + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + collectUnresolvedBundles(childGroup, bundleLookup, missingBundles, missingProcessorTypes, missingControllerServiceTypes); + } + } + } + + private String formatBundle(final Bundle bundle) { + return "%s:%s:%s".formatted(bundle.getGroup(), bundle.getArtifact(), bundle.getVersion()); + } + + private boolean isBundleResolvable(final String componentType, final Bundle currentBundle, final ComponentBundleLookup bundleLookup) { + final List availableBundles = bundleLookup.getAvailableBundles(componentType); + + if (availableBundles.contains(currentBundle)) { + return true; + } + + // With RESOLVE_BUNDLE, a bundle can be resolved only if exactly one alternative bundle is available + return availableBundles.size() == 1; + } + + private void recreateWorkingFlowContext() { + destroyWorkingContext(); + workingFlowContext = flowContextFactory.createWorkingFlowContext(identifier, + connectorDetails.getComponentLog(), activeFlowContext.getConfigurationContext(), activeFlowContext.getBundle()); + + getComponentLog().info("Working Flow Context has been recreated"); + } + + @Override + public void pauseValidationTrigger() { + triggerValidation = false; + } + + @Override + public void resumeValidationTrigger() { + triggerValidation = true; + logger.debug("Resuming Triggering of Validation State for {}; Resetting validation state", this); + resetValidationState(); + } + + @Override + public boolean isValidationPaused() { + return !triggerValidation; + } + + @Override + public List verifyConfigurationStep(final String stepName, final StepConfiguration configurationOverrides) { + logger.debug("Verifying configuration step {} for {}", stepName, this); + final List invalidSecretRefs = new ArrayList<>(); + final List invalidAssetRefs = new ArrayList<>(); + final Map resolvedPropertyOverrides = resolvePropertyReferences(configurationOverrides, invalidSecretRefs, invalidAssetRefs); + + final List results = new ArrayList<>(); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + + final DescribedValueProvider allowableValueProvider = (step, propertyName) -> fetchAllowableValues(step, propertyName, workingFlowContext); + + final MutableConnectorConfigurationContext configContext = workingFlowContext.getConfigurationContext().createWithOverrides(stepName, resolvedPropertyOverrides); + final ConnectorConfiguration connectorConfig = configContext.toConnectorConfiguration(); + final ParameterContextFacade paramContext = workingFlowContext.getParameterContext(); + final ConnectorValidationContext validationContext = new StandardConnectorValidationContext(connectorConfig, allowableValueProvider, paramContext); + + final Optional optionalStep = getConfigurationStep(stepName); + if (optionalStep.isEmpty()) { + results.add(new ConfigVerificationResult.Builder() + .verificationStepName("Property Validation") + .outcome(Outcome.FAILED) + .explanation("Configuration step with name '" + stepName + "' does not exist.") + .build()); + return results; + } + + final ConfigurationStep configurationStep = optionalStep.get(); + + final List validationResults = new ArrayList<>(); + validatePropertyReferences(configurationStep, configurationOverrides, validationResults); + + // If there are any invalid secrets or assets referenced, add Validation Results for them. + addInvalidReferenceResults(validationResults, invalidSecretRefs, invalidAssetRefs); + + // If there are any framework-level validation failures, we do not run the Connector-specific validation because + // doing so would mean that we must provide weak guarantees about the state of the configuration when the Connector's + // validation is invoked. But if there are no framework-level validation failures, we can proceed to invoke the + // Connector's validation logic. + if (validationResults.isEmpty()) { + final List implValidationResults = getConnector().validateConfigurationStep(configurationStep, configContext, validationContext); + validationResults.addAll(implValidationResults); + } + + final List invalidConfigResults = validationResults.stream() + .filter(result -> !result.isValid()) + .map(this::createConfigVerificationResult) + .toList(); + + if (invalidConfigResults.isEmpty()) { + results.add(new ConfigVerificationResult.Builder() + .verificationStepName("Property Validation") + .outcome(Outcome.SUCCESSFUL) + .build()); + + results.addAll(getConnector().verifyConfigurationStep(stepName, resolvedPropertyOverrides, workingFlowContext)); + } else { + results.addAll(invalidConfigResults); + } + + logger.debug("Completed verification of configuration step {} for {}", stepName, this); + return results; + } + } + + private ConfigVerificationResult createConfigVerificationResult(final ValidationResult validationResult) { + return new ConfigVerificationResult.Builder() + .verificationStepName("Property Validation - " + validationResult.getSubject()) + .outcome(validationResult.isValid() ? Outcome.SUCCESSFUL : Outcome.FAILED) + .subject(validationResult.getSubject()) + .explanation(validationResult.getExplanation()) + .build(); + } + + private Map resolvePropertyReferences(final StepConfiguration configurationOverrides, final List invalidSecretRefs, + final List invalidAssetRefs) { + + final Map resolvedProperties = new HashMap<>(); + + try { + // Secret References can be expensive to lookup so we don't want to call getSecret() for each one. Instead, we + // want to find all Secrets by Provider and then call fetchSecrets() once per Provider. + final Set secretReferences = configurationOverrides.getPropertyValues().values().stream() + .filter(Objects::nonNull) + .filter(ref -> ref.getValueType() == ConnectorValueType.SECRET_REFERENCE) + .map(ref -> (SecretReference) ref) + .collect(Collectors.toSet()); + + final Map secretsByReference = initializationContext.getSecretsManager().getSecrets(secretReferences); + secretsByReference.forEach((ref, secret) -> { + if (secret == null) { + invalidSecretRefs.add(ref); + } + }); + + for (final Map.Entry entry : configurationOverrides.getPropertyValues().entrySet()) { + final String propertyName = entry.getKey(); + final ConnectorValueReference valueReference = entry.getValue(); + + if (valueReference == null) { + continue; + } + + // We've already looked up secrets above, so use the cached value here. + if (valueReference.getValueType() == ConnectorValueType.SECRET_REFERENCE) { + final SecretReference secretReference = (SecretReference) valueReference; + final Secret secret = secretsByReference.get(secretReference); + final String resolvedValue = (secret == null) ? null : secret.getValue(); + resolvedProperties.put(propertyName, resolvedValue); + continue; + } + + final String resolvedValue = resolvePropertyReference(valueReference); + resolvedProperties.put(propertyName, resolvedValue); + + if (resolvedValue == null && valueReference.getValueType() == ConnectorValueType.ASSET_REFERENCE) { + invalidAssetRefs.add((AssetReference) valueReference); + } + } + } catch (final IOException ioe) { + throw new UncheckedIOException("Failed to resolve Secret references for " + this, ioe); + } + + return resolvedProperties; + } + + private String resolvePropertyReference(final ConnectorValueReference valueReference) throws IOException { + if (valueReference == null) { + return null; + } + + return switch (valueReference) { + case StringLiteralValue stringLiteralValue -> stringLiteralValue.getValue(); + case AssetReference assetReference -> resolveAssetReferences(assetReference); + case SecretReference secretReference -> initializationContext.getSecretsManager() + .getSecret(secretReference) + .map(Secret::getValue) + .orElse(null); + }; + } + + private String resolveAssetReferences(final AssetReference assetReference) { + final Set resolvedAssetValues = new HashSet<>(); + for (final String assetId : assetReference.getAssetIdentifiers()) { + initializationContext.getAssetManager().getAsset(assetId) + .map(Asset::getFile) + .map(File::getAbsolutePath) + .ifPresent(resolvedAssetValues::add); + } + + logger.debug("Resolved {} to {} for {}", assetReference, resolvedAssetValues, this); + return String.join(",", resolvedAssetValues); + } + + private Optional getConfigurationStep(final String stepName) { + for (final ConfigurationStep step : getConfigurationSteps()) { + if (Objects.equals(step.getName(), stepName)) { + return Optional.of(step); + } + } + + return Optional.empty(); + } + + @Override + public List verify() { + logger.debug("Verifying {}", this); + final List results = new ArrayList<>(); + + final ValidationState state = performValidation(); + if (state.getStatus() == ValidationStatus.INVALID) { + final List validationFailureExplanations = state.getValidationErrors().stream() + .map(ValidationResult::getExplanation) + .toList(); + + results.add(new ConfigVerificationResult.Builder() + .verificationStepName("Property Validation") + .outcome(Outcome.FAILED) + .explanation("There are " + validationFailureExplanations.size() + " validation failures: " + validationFailureExplanations) + .build()); + + logger.debug("Completed verification for {} with validation failures", this); + return results; + } + + workingFlowContext.getConfigurationContext().resolvePropertyValues(); + + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + results.addAll(getConnector().verify(workingFlowContext)); + } + + logger.debug("Completed verification for {}", this); + return results; + } + + + @Override + public String getIdentifier() { + return identifier; + } + + @Override + public String getProcessGroupIdentifier() { + return null; + } + + @Override + public ComponentLog getComponentLog() { + return connectorDetails.getComponentLog(); + } + + @Override + public List getConfigurationSteps() { + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + return getConnector().getConfigurationSteps(); + } + } + + @Override + public FrameworkFlowContext getActiveFlowContext() { + return activeFlowContext; + } + + @Override + public FrameworkFlowContext getWorkingFlowContext() { + return workingFlowContext; + } + + @Override + public void discardWorkingConfiguration() { + recreateWorkingFlowContext(); + logger.debug("Discarded working configuration for {}", this); + } + + @Override + public List getAvailableActions() { + final List actions = new ArrayList<>(); + final ConnectorState currentState = getCurrentState(); + final boolean dataQueued = activeFlowContext.getManagedProcessGroup().isDataQueued(); + final boolean stopped = isStopped(); + + actions.add(createStartAction(stopped)); + actions.add(createStopAction(currentState)); + actions.add(createConfigureAction()); + actions.add(createDiscardWorkingConfigAction()); + actions.add(createPurgeFlowFilesAction(stopped, dataQueued)); + actions.add(createDrainFlowFilesAction(stopped, dataQueued)); + actions.add(createCancelDrainFlowFilesAction(currentState == ConnectorState.DRAINING)); + actions.add(createApplyUpdatesAction(currentState)); + actions.add(createDeleteAction(stopped, dataQueued)); + + return actions; + } + + private boolean isStopped() { + final ConnectorState currentState = getCurrentState(); + if (currentState == ConnectorState.STOPPED) { + return true; + } + if (currentState == ConnectorState.UPDATED || currentState == ConnectorState.UPDATE_FAILED) { + return !hasActiveThread(getActiveFlowContext().getManagedProcessGroup()); + } + + return false; + } + + private ConnectorAction createStartAction(final boolean stopped) { + final boolean allowed; + final String reason; + + if (!stopped) { + allowed = false; + reason = "Connector is not stopped"; + } else { + final Collection validationResults = getValidationErrors(); + if (validationResults.isEmpty()) { + allowed = true; + reason = null; + } else { + allowed = false; + reason = "Connector is not valid: " + validationResults.stream() + .map(ValidationResult::getExplanation) + .collect(Collectors.joining("; ")); + } + } + + return new StandardConnectorAction("START", "Start the connector", allowed, reason); + } + + private ConnectorAction createStopAction(final ConnectorState currentState) { + final boolean allowed; + if (currentState == ConnectorState.RUNNING || currentState == ConnectorState.STARTING) { + allowed = true; + } else if (currentState == ConnectorState.UPDATED || currentState == ConnectorState.UPDATE_FAILED) { + allowed = hasActiveThread(activeFlowContext.getManagedProcessGroup()); + } else { + allowed = false; + } + + final String reason = allowed ? null : "Connector is not running"; + return new StandardConnectorAction("STOP", "Stop the connector", allowed, reason); + } + + private ConnectorAction createConfigureAction() { + return new StandardConnectorAction("CONFIGURE", "Configure the connector", true, null); + } + + private ConnectorAction createDiscardWorkingConfigAction() { + final boolean allowed = hasWorkingConfigurationChanges(); + final String reason = allowed ? null : "No pending changes to discard"; + + return new StandardConnectorAction("DISCARD_WORKING_CONFIGURATION", "Discard any changes made to the working configuration", allowed, reason); + } + + private boolean hasActiveThread(final ProcessGroup group) { + for (final ProcessorNode processor : group.getProcessors()) { + if (processor.getActiveThreadCount() > 0) { + return true; + } + } + + for (final ProcessGroup childGroup : group.getProcessGroups()) { + if (hasActiveThread(childGroup)) { + return true; + } + } + + return false; + } + + private ConnectorAction createPurgeFlowFilesAction(final boolean stopped, final boolean dataQueued) { + return createDataQueuedAction(stopped, dataQueued, "PURGE_FLOWFILES", "Purge all FlowFiles from the connector, dropping all data without processing it"); + } + + private ConnectorAction createDrainFlowFilesAction(final boolean stopped, final boolean dataQueued) { + return createDataQueuedAction(stopped, dataQueued, "DRAIN_FLOWFILES", "Process data that is currently in the flow but do not ingest any additional data"); + } + + private static ConnectorAction createDataQueuedAction(final boolean stopped, final boolean dataQueued, final String actionName, final String description) { + final boolean allowed; + final String reason; + + if (!stopped) { + allowed = false; + reason = "Connector must be stopped"; + } else if (!dataQueued) { + allowed = false; + reason = "No data is queued"; + } else { + allowed = true; + reason = null; + } + + return new StandardConnectorAction(actionName, description, allowed, reason); + } + + private ConnectorAction createCancelDrainFlowFilesAction(final boolean draining) { + if (draining) { + return new StandardConnectorAction("CANCEL_DRAIN_FLOWFILES", "Cancel the ongoing drain of FlowFiles", true, null); + } + + return new StandardConnectorAction("CANCEL_DRAIN_FLOWFILES", "Cancel the ongoing drain of FlowFiles", false, + "Connector is not currently draining FlowFiles"); + } + + private ConnectorAction createApplyUpdatesAction(final ConnectorState currentState) { + final boolean allowed; + final String reason; + + if (currentState == ConnectorState.PREPARING_FOR_UPDATE || currentState == ConnectorState.UPDATING) { + allowed = false; + reason = "Connector is updating"; + } else if (!hasWorkingConfigurationChanges()) { + allowed = false; + reason = "No pending changes"; + } else { + allowed = true; + reason = null; + } + + return new StandardConnectorAction("APPLY_UPDATES", "Apply the working configuration to the active configuration", allowed, reason); + } + + private ConnectorAction createDeleteAction(final boolean stopped, final boolean dataQueued) { + final boolean allowed; + final String reason; + + if (!stopped) { + allowed = false; + reason = "Connector must be stopped"; + } else if (dataQueued) { + allowed = false; + reason = "Data is queued"; + } else { + allowed = true; + reason = null; + } + + return new StandardConnectorAction("DELETE", "Delete the connector", allowed, reason); + } + + private boolean hasWorkingConfigurationChanges() { + final FrameworkFlowContext workingContext = this.workingFlowContext; + if (workingContext == null) { + return false; + } + + final ConnectorConfiguration activeConfig = activeFlowContext.getConfigurationContext().toConnectorConfiguration(); + final ConnectorConfiguration workingConfig = workingContext.getConfigurationContext().toConnectorConfiguration(); + return !Objects.equals(activeConfig, workingConfig); + } + + @Override + public Authorizable getParentAuthorizable() { + return parentAuthorizable; + } + + @Override + public Resource getResource() { + return ResourceFactory.getComponentResource(ResourceType.Connector, getIdentifier(), getName()); + } + + @Override + public Optional getVersionedComponentId() { + return Optional.ofNullable(versionedComponentId.get()); + } + + @Override + public void setVersionedComponentId(final String versionedComponentId) { + boolean updated = false; + while (!updated) { + final String currentId = this.versionedComponentId.get(); + + if (currentId == null) { + updated = this.versionedComponentId.compareAndSet(null, versionedComponentId); + } else if (currentId.equals(versionedComponentId)) { + return; + } else if (versionedComponentId == null) { + updated = this.versionedComponentId.compareAndSet(currentId, null); + } else { + throw new IllegalStateException(this + " is already under version control"); + } + } + } + + private void resetValidationState() { + validationState.set(new ValidationState(ValidationStatus.VALIDATING, Collections.emptyList())); + validationTrigger.triggerAsync(this); + logger.debug("Validation state has been reset for {}", this); + } + + @Override + public ValidationStatus getValidationStatus() { + return validationState.get().getStatus(); + } + + @Override + public ValidationState getValidationState() { + return validationState.get(); + } + + @Override + public Collection getValidationErrors() { + final ValidationState state = validationState.get(); + return state.getValidationErrors(); + } + + @Override + public ValidationState performValidation() { + logger.debug("Performing validation for {}", this); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + + final List allResults = new ArrayList<>(); + + if (unresolvedBundleValidationResult != null) { + allResults.add(unresolvedBundleValidationResult); + } else { + final ConnectorValidationContext validationContext = createValidationContext(activeFlowContext); + + validateManagedFlowComponents(allResults); + validatePropertyReferences(allResults); + + if (allResults.isEmpty()) { + try { + final List implValidationResults = getConnector().validate(activeFlowContext, validationContext); + allResults.addAll(implValidationResults); + } catch (final Exception e) { + allResults.add(new ValidationResult.Builder() + .subject("Validation Failure") + .valid(false) + .explanation("Encountered a failure while attempting to perform validation: " + e.getMessage()) + .build()); + } + } + } + + final ValidationState resultState; + if (allResults.isEmpty()) { + resultState = new ValidationState(ValidationStatus.VALID, Collections.emptyList()); + } else { + // Filter out any results that are 'valid' and any results that are invalid due to the fact that a Controller Service is disabled, + // since these will not be relevant when started. + final List relevantResults = allResults.stream() + .filter(result -> !result.isValid()) + .filter(result -> !DisabledServiceValidationResult.isMatch(result)) + .toList(); + + if (relevantResults.isEmpty()) { + resultState = new ValidationState(ValidationStatus.VALID, Collections.emptyList()); + } else { + resultState = new ValidationState(ValidationStatus.INVALID, relevantResults); + } + } + + validationState.set(resultState); + + if (resultState.getStatus() == ValidationStatus.VALID) { + logger.info("Validation completed for {}. Connector is valid.", this); + } else { + logger.info("Validation completed for {}. Connector is invalid: {}", this, + resultState.getValidationErrors().stream() + .map(ValidationResult::getExplanation) + .collect(Collectors.joining("; "))); + } + + return resultState; + } + } + + @Override + public void validateComponents(final ValidationTrigger validationTrigger) { + final ProcessGroup managedGroup = activeFlowContext.getManagedProcessGroup(); + + for (final ProcessorNode processor : managedGroup.findAllProcessors()) { + validationTrigger.trigger(processor); + } + + for (final ControllerServiceNode service : managedGroup.findAllControllerServices()) { + validationTrigger.trigger(service); + } + } + + private void validateManagedFlowComponents(final List results) { + final ProcessGroup managedProcessGroup = activeFlowContext.getManagedProcessGroup(); + + // Check for any missing / ghosted Processors + final Set missingProcessors = new HashSet<>(); + for (final ProcessorNode processor : managedProcessGroup.findAllProcessors()) { + if (processor.isExtensionMissing()) { + missingProcessors.add(getSimpleClassName(processor.getCanonicalClassName()) + " from NAR " + processor.getBundleCoordinate()); + } + } + + if (!missingProcessors.isEmpty()) { + results.add(new ValidationResult.Builder() + .subject("Missing Processors") + .valid(false) + .explanation("The following processors are missing: " + missingProcessors) + .build()); + } + + // Check for any missing / ghosted Controller Services + final Set missingControllerServices = new HashSet<>(); + for (final ControllerServiceNode controllerService : managedProcessGroup.findAllControllerServices()) { + if (controllerService.isExtensionMissing()) { + missingControllerServices.add(getSimpleClassName(controllerService.getCanonicalClassName()) + " from NAR " + controllerService.getBundleCoordinate()); + } + } + + if (!missingControllerServices.isEmpty()) { + results.add(new ValidationResult.Builder() + .subject("Missing Controller Services") + .valid(false) + .explanation("The following controller services are missing: " + missingControllerServices) + .build()); + } + } + + private static String getSimpleClassName(final String canonicalClassName) { + final int lastDot = canonicalClassName.lastIndexOf('.'); + return lastDot < 0 ? canonicalClassName : canonicalClassName.substring(lastDot + 1); + } + + private void validatePropertyReferences(final List allResults) { + final List configurationSteps = getConnector().getConfigurationSteps(); + final ConnectorConfiguration connectorConfiguration = activeFlowContext.getConfigurationContext().toConnectorConfiguration(); + + for (final ConfigurationStep step : configurationSteps) { + final NamedStepConfiguration namedStepConfig = connectorConfiguration.getNamedStepConfiguration(step.getName()); + if (namedStepConfig == null) { + continue; + } + + final StepConfiguration stepConfiguration = namedStepConfig.configuration(); + validatePropertyReferences(step, namedStepConfig.configuration(), allResults); + + // Check for invalid Secret and Asset references + final List invalidSecrets = new ArrayList<>(); + final List invalidAssets = new ArrayList<>(); + resolvePropertyReferences(stepConfiguration, invalidSecrets, invalidAssets); + addInvalidReferenceResults(allResults, invalidSecrets, invalidAssets); + } + } + + private void addInvalidReferenceResults(final List results, final List invalidSecretRefs, final List invalidAssetRefs) { + for (final SecretReference invalidSecretRef : invalidSecretRefs) { + results.add(new ValidationResult.Builder() + .subject("Secret Reference") + .valid(false) + .explanation("The referenced secret [" + invalidSecretRef.getFullyQualifiedName() + "] could not be found") + .build()); + } + + for (final AssetReference invalidAssetRef : invalidAssetRefs) { + results.add(new ValidationResult.Builder() + .subject("Asset Reference") + .valid(false) + .explanation("The referenced assets [" + StringUtils.join(invalidAssetRef.getAssetIdentifiers(), ",") + "] could not be found") + .build()); + } + } + + private void validatePropertyReferences(final ConfigurationStep step, final StepConfiguration stepConfig, final List allResults) { + for (final ConnectorPropertyGroup propertyGroup : step.getPropertyGroups()) { + for (final ConnectorPropertyDescriptor descriptor : propertyGroup.getProperties()) { + final PropertyType propertyType = descriptor.getType(); + final ConnectorValueReference reference = stepConfig.getPropertyValue(descriptor.getName()); + + final String subject = step.getName() + " / " + descriptor.getName(); + + if (!isReferenceAllowed(reference, propertyType)) { + final String providedReferenceType = switch (reference.getValueType()) { + case ASSET_REFERENCE -> ""; + case SECRET_REFERENCE -> ""; + case STRING_LITERAL -> ""; + }; + + final String expectedReferenceType = propertyType == PropertyType.SECRET ? "a Secret reference" : "an Explicit value"; + + allResults.add(new ValidationResult.Builder() + .subject(subject) + .input(providedReferenceType) + .explanation("This property must be configured with " + expectedReferenceType) + .build()); + } + } + } + } + + private boolean isReferenceAllowed(final ConnectorValueReference reference, final PropertyType propertyType) { + // If the reference is null or its value is unset, then it is allowed + if (reference == null) { + return true; + } + + switch (reference) { + case StringLiteralValue stringLiteralValue -> { + if (stringLiteralValue.getValue() == null) { + return true; + } + } + case AssetReference assetReference -> { + if (assetReference.getAssetIdentifiers() == null || assetReference.getAssetIdentifiers().isEmpty()) { + return true; + } + } + case SecretReference secretReference -> { + if (secretReference.getSecretName() == null) { + return true; + } + } + } + + if (propertyType == PropertyType.SECRET) { + return reference.getValueType() == ConnectorValueType.SECRET_REFERENCE; + } + + if (propertyType == PropertyType.ASSET || propertyType == PropertyType.ASSET_LIST) { + return reference.getValueType() == ConnectorValueType.ASSET_REFERENCE; + } + + return reference.getValueType() != ConnectorValueType.SECRET_REFERENCE && reference.getValueType() != ConnectorValueType.ASSET_REFERENCE; + } + + private ConnectorValidationContext createValidationContext(final FrameworkFlowContext context) { + final DescribedValueProvider allowableValueProvider = (stepName, propertyName) -> + fetchAllowableValues(stepName, propertyName, context); + final ConnectorConfiguration connectorConfiguration = context.getConfigurationContext().toConnectorConfiguration(); + return new StandardConnectorValidationContext(connectorConfiguration, allowableValueProvider, context.getParameterContext()); + } + + private List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext context) { + final List allowableValues; + try (NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, getConnector().getClass(), getIdentifier())) { + allowableValues = getConnector().fetchAllowableValues(stepName, propertyName, context); + } + + if (allowableValues == null || allowableValues.isEmpty()) { + return Collections.emptyList(); + } + + return allowableValues; + } + + @Override + public boolean equals(final Object o) { + if (o == null || getClass() != o.getClass()) { + return false; + } + final StandardConnectorNode that = (StandardConnectorNode) o; + return Objects.equals(identifier, that.identifier); + } + + @Override + public int hashCode() { + return Objects.hashCode(identifier); + } + + @Override + public String toString() { + return "StandardConnectorNode[id=" + identifier + ", name=" + name + ", state=" + stateTransition.getCurrentState() + "]"; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorPropertyValue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorPropertyValue.java new file mode 100644 index 000000000000..82603da0b8cb --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorPropertyValue.java @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.util.FormatUtils; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; + +public class StandardConnectorPropertyValue implements ConnectorPropertyValue { + + private final String rawValue; + + public StandardConnectorPropertyValue(final String rawValue) { + this.rawValue = rawValue; + } + + @Override + public String getValue() { + return rawValue; + } + + @Override + public Integer asInteger() { + return rawValue == null ? null : Integer.parseInt(rawValue); + } + + @Override + public Long asLong() { + return rawValue == null ? null : Long.parseLong(rawValue); + } + + @Override + public Boolean asBoolean() { + return rawValue == null ? null : Boolean.parseBoolean(rawValue); + } + + @Override + public Float asFloat() { + return rawValue == null ? null : Float.parseFloat(rawValue); + } + + @Override + public Double asDouble() { + return rawValue == null ? null : Double.parseDouble(rawValue); + } + + @Override + public Long asTimePeriod(final TimeUnit timeUnit) { + return rawValue == null ? null : FormatUtils.getTimeDuration(rawValue, timeUnit); + } + + @Override + public Duration asDuration() { + return rawValue == null ? null : Duration.ofNanos(asTimePeriod(TimeUnit.NANOSECONDS)); + } + + @Override + public Double asDataSize(final DataUnit dataUnit) { + return rawValue == null ? null : DataUnit.parseDataSize(rawValue.trim(), dataUnit); + } + + @Override + public List asList() { + if (rawValue == null) { + return Collections.emptyList(); + } + + final String[] splits = rawValue.split(","); + final List values = new ArrayList<>(splits.length); + for (final String split : splits) { + values.add(split.trim()); + } + + return values; + } + + @Override + public boolean isSet() { + return rawValue != null; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepoInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepoInitializationContext.java new file mode 100644 index 000000000000..a0661fa58f92 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepoInitializationContext.java @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.controller.NodeTypeProvider; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.nar.ExtensionManager; + +public class StandardConnectorRepoInitializationContext implements ConnectorRepositoryInitializationContext { + private final FlowManager flowManager; + private final ExtensionManager extensionManager; + private final SecretsManager secretsManager; + private final AssetManager assetManager; + private final NodeTypeProvider nodeTypeProvider; + private final ConnectorRequestReplicator requestReplicator; + private final ConnectorConfigurationProvider connectorConfigurationProvider; + + public StandardConnectorRepoInitializationContext(final FlowManager flowManager, + final ExtensionManager extensionManager, + final SecretsManager secretsManager, + final AssetManager assetManager, + final NodeTypeProvider nodeTypeProvider, + final ConnectorRequestReplicator requestReplicator, + final ConnectorConfigurationProvider connectorConfigurationProvider) { + this.flowManager = flowManager; + this.extensionManager = extensionManager; + this.secretsManager = secretsManager; + this.assetManager = assetManager; + this.nodeTypeProvider = nodeTypeProvider; + this.requestReplicator = requestReplicator; + this.connectorConfigurationProvider = connectorConfigurationProvider; + } + + @Override + public FlowManager getFlowManager() { + return flowManager; + } + + @Override + public ExtensionManager getExtensionManager() { + return extensionManager; + } + + @Override + public SecretsManager getSecretsManager() { + return secretsManager; + } + + @Override + public AssetManager getAssetManager() { + return assetManager; + } + + @Override + public NodeTypeProvider getNodeTypeProvider() { + return nodeTypeProvider; + } + + @Override + public ConnectorRequestReplicator getRequestReplicator() { + return requestReplicator; + } + + @Override + public ConnectorConfigurationProvider getConnectorConfigurationProvider() { + return connectorConfigurationProvider; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java new file mode 100644 index 000000000000..2df018c63825 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java @@ -0,0 +1,594 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.annotation.lifecycle.OnRemoved; +import org.apache.nifi.asset.Asset; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.engine.FlowEngine; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConfigurationStep; +import org.apache.nifi.flow.VersionedConnectorValueReference; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.nar.NarCloseable; +import org.apache.nifi.util.ReflectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.InputStream; +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; + +public class StandardConnectorRepository implements ConnectorRepository { + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorRepository.class); + + private final Map connectors = new ConcurrentHashMap<>(); + private final FlowEngine lifecycleExecutor = new FlowEngine(8, "NiFi Connector Lifecycle"); + + private volatile ExtensionManager extensionManager; + private volatile ConnectorRequestReplicator requestReplicator; + private volatile SecretsManager secretsManager; + private volatile AssetManager assetManager; + private volatile ConnectorConfigurationProvider configurationProvider; + + @Override + public void initialize(final ConnectorRepositoryInitializationContext context) { + logger.debug("Initializing ConnectorRepository"); + this.extensionManager = context.getExtensionManager(); + this.requestReplicator = context.getRequestReplicator(); + this.secretsManager = context.getSecretsManager(); + this.assetManager = context.getAssetManager(); + this.configurationProvider = context.getConnectorConfigurationProvider(); + logger.debug("Successfully initialized ConnectorRepository with configurationProvider={}", configurationProvider != null ? configurationProvider.getClass().getSimpleName() : "null"); + } + + @Override + public void verifyCreate(final String connectorId) { + if (connectors.containsKey(connectorId)) { + throw new IllegalStateException("A Connector already exists with ID %s".formatted(connectorId)); + } + if (configurationProvider != null) { + configurationProvider.verifyCreate(connectorId); + } + } + + @Override + public void addConnector(final ConnectorNode connector) { + syncFromProvider(connector); + connectors.put(connector.getIdentifier(), connector); + } + + @Override + public void restoreConnector(final ConnectorNode connector) { + connectors.put(connector.getIdentifier(), connector); + logger.debug("Successfully restored {}", connector); + } + + @Override + public void removeConnector(final String connectorId) { + logger.debug("Removing {}", connectorId); + final ConnectorNode connectorNode = connectors.get(connectorId); + if (connectorNode == null) { + throw new IllegalStateException("No connector found with ID " + connectorId); + } + + connectorNode.verifyCanDelete(); + if (configurationProvider != null) { + configurationProvider.delete(connectorId); + } + connectors.remove(connectorId); + + final Class taskClass = connectorNode.getConnector().getClass(); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, taskClass, connectorId)) { + ReflectionUtils.quietlyInvokeMethodsWithAnnotation(OnRemoved.class, connectorNode.getConnector()); + } + + extensionManager.removeInstanceClassLoader(connectorId); + } + + @Override + public ConnectorNode getConnector(final String identifier) { + final ConnectorNode connector = connectors.get(identifier); + if (connector != null) { + syncFromProvider(connector); + } + return connector; + } + + @Override + public List getConnectors() { + final List connectorList = List.copyOf(connectors.values()); + for (final ConnectorNode connector : connectorList) { + syncFromProvider(connector); + } + return connectorList; + } + + @Override + public Future startConnector(final ConnectorNode connector) { + return connector.start(lifecycleExecutor); + } + + @Override + public Future stopConnector(final ConnectorNode connector) { + return connector.stop(lifecycleExecutor); + } + + @Override + public Future restartConnector(final ConnectorNode connector) { + final CompletableFuture restartCompleteFuture = new CompletableFuture<>(); + restartConnector(connector, restartCompleteFuture); + return restartCompleteFuture; + } + + private void restartConnector(final ConnectorNode connector, final CompletableFuture restartCompleteFuture) { + try { + final Future stopFuture = connector.stop(lifecycleExecutor); + stopFuture.get(); + + final Future startFuture = connector.start(lifecycleExecutor); + startFuture.get(); + + logger.info("Successfully restarted connector [{}]", connector.getIdentifier()); + restartCompleteFuture.complete(null); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + logger.error("Interrupted while restarting connector [{}]", connector.getIdentifier(), e); + restartCompleteFuture.completeExceptionally(e); + } catch (final ExecutionException e) { + logger.error("Failed to restart connector [{}]", connector.getIdentifier(), e.getCause()); + restartCompleteFuture.completeExceptionally(e); + } + } + + @Override + public void applyUpdate(final ConnectorNode connector, final ConnectorUpdateContext context) throws FlowUpdateException { + logger.debug("Applying update to {}", connector); + + if (configurationProvider != null && !configurationProvider.shouldApplyUpdate(connector.getIdentifier())) { + logger.info("ConnectorConfigurationProvider indicated framework should not apply update for {}; skipping framework update process", connector); + return; + } + + final ConnectorState initialDesiredState = connector.getDesiredState(); + logger.info("Applying update to Connector {}", connector); + + // Sync asset binaries first (provider downloads changed assets, assigns new UUIDs), + // then load the updated configuration (which will reflect any new NiFi UUIDs). + syncAssetsFromProvider(connector); + + // Transition the connector's state to PREPARING_FOR_UPDATE before starting the background process. + // This allows us to ensure that if we poll and see the state in the same state it was in before that + // we know the update has already completed (successfully or otherwise). + logger.debug("Transitioning {} to PREPARING_FOR_UPDATE state before applying update", connector); + connector.transitionStateForUpdating(); + logger.debug("{} is now in PREPARING_FOR_UPDATE state", connector); + + // Update connector in a background thread. This will handle transitioning the Connector state appropriately + // so that it's clear when the update has completed. + lifecycleExecutor.submit(() -> { + updateConnector(connector, initialDesiredState, context); + cleanUpAssets(connector); + }); + } + + private void updateConnector(final ConnectorNode connector, final ConnectorState initialDesiredState, final ConnectorUpdateContext context) { + logger.debug("Updating {}", connector); + try { + // Perform whatever preparation is necessary for the update. Default implementation is to stop the connector. + logger.debug("Preparing {} for update", connector); + connector.prepareForUpdate(); + + // Wait for Connector State to become UPDATING + logger.debug("Waiting for {} to transition to UPDATING state", connector); + waitForState(connector, Set.of(ConnectorState.UPDATING), Set.of(ConnectorState.PREPARING_FOR_UPDATE)); + + // Apply the update to the connector. + logger.info("{} has now completed preparations for update; applying update now", connector); + connector.applyUpdate(); + logger.info("{} has successfully applied update", connector); + + // Now that the update has been applied, save the flow so that the updated configuration is persisted. + context.saveFlow(); + + // Wait for all nodes to complete the update. + waitForState(connector, Set.of(ConnectorState.UPDATED), Set.of(ConnectorState.UPDATING)); + logger.info("{} has successfully completed update on all nodes", connector); + + // If the initial desired state was RUNNING, start the connector again. Otherwise, stop it. + // We don't simply leave it be as the prepareForUpdate / update may have changed the state of some components. + if (initialDesiredState == ConnectorState.RUNNING) { + logger.info("{} has been successfully updated; starting to resume initial state", connector); + connector.start(lifecycleExecutor); + } else { + logger.info("{} has been successfully updated; stopping to resume initial state", connector); + connector.stop(lifecycleExecutor); + } + + // We've updated the state of the connector so save flow again + context.saveFlow(); + } catch (final Exception e) { + logger.error("Failed to apply update for {}", connector, e); + connector.abortUpdate(e); + } + } + + private void waitForState(final ConnectorNode connector, final Set desiredStates, final Set allowableStates) + throws FlowUpdateException, IOException, InterruptedException { + + // Wait for Connector State to become the desired state + int iterations = 0; + final long startNanos = System.nanoTime(); + while (true) { + final ConnectorState clusterState = requestReplicator.getState(connector.getIdentifier()); + if (desiredStates.contains(clusterState)) { + logger.info("State for {} is now {}", connector, clusterState); + break; + } else if (allowableStates.contains(clusterState)) { + final long elapsedSeconds = Duration.ofNanos(System.nanoTime() - startNanos).toSeconds(); + if (++iterations % 10 == 0) { + logger.info("Waiting for {} to transition to one of {}. Current state is {}; elapsed time = {} secs", connector, desiredStates, clusterState, elapsedSeconds); + } else { + logger.debug("Waiting for {} to transition to one of {}. Current state is {}; elapsed time = {} secs", connector, desiredStates, clusterState, elapsedSeconds); + } + + Thread.sleep(Duration.ofSeconds(1)); + continue; + } else if (clusterState == ConnectorState.UPDATE_FAILED) { + throw new FlowUpdateException("State of " + connector + " transitioned to UPDATE_FAILED while waiting for state to become one of " + desiredStates); + } + + throw new FlowUpdateException("While waiting for %s to transition to state in set %s, connector transitioned to unexpected state: %s".formatted(connector, desiredStates, clusterState)); + } + } + + private void cleanUpAssets(final ConnectorNode connector) { + final String connectorId = connector.getIdentifier(); + final Set referencedAssetIds = new HashSet<>(); + collectReferencedAssetIds(connector.getActiveFlowContext(), referencedAssetIds); + collectReferencedAssetIds(connector.getWorkingFlowContext(), referencedAssetIds); + + logger.debug("Found {} assets referenced for Connector [{}]", referencedAssetIds.size(), connectorId); + + final List allConnectorAssets = assetManager.getAssets(connectorId); + for (final Asset asset : allConnectorAssets) { + final String assetId = asset.getIdentifier(); + if (!referencedAssetIds.contains(assetId)) { + try { + logger.info("Deleting unreferenced asset [id={},name={}] for connector [{}]", assetId, asset.getName(), connectorId); + + if (configurationProvider != null) { + // Provider deletes from external store and local AssetManager + configurationProvider.deleteAsset(connectorId, assetId); + } else { + assetManager.deleteAsset(assetId); + } + } catch (final Exception e) { + logger.warn("Unable to delete unreferenced asset [id={},name={}] for connector [{}]", assetId, asset.getName(), connectorId, e); + } + } + } + } + + private void collectReferencedAssetIds(final FrameworkFlowContext flowContext, final Set referencedAssetIds) { + if (flowContext == null) { + return; + } + + final ConnectorConfiguration configuration = flowContext.getConfigurationContext().toConnectorConfiguration(); + for (final NamedStepConfiguration namedStepConfiguration : configuration.getNamedStepConfigurations()) { + final StepConfiguration stepConfiguration = namedStepConfiguration.configuration(); + final Map stepPropertyValues = stepConfiguration.getPropertyValues(); + if (stepPropertyValues == null) { + continue; + } + for (final ConnectorValueReference valueReference : stepPropertyValues.values()) { + if (valueReference instanceof AssetReference assetReference) { + referencedAssetIds.addAll(assetReference.getAssetIdentifiers()); + } + } + } + } + + @Override + public void updateConnector(final ConnectorNode connector, final String name) { + if (configurationProvider != null) { + final ConnectorWorkingConfiguration workingConfiguration = buildWorkingConfiguration(connector); + workingConfiguration.setName(name); + configurationProvider.save(connector.getIdentifier(), workingConfiguration); + } + connector.setName(name); + } + + @Override + public void configureConnector(final ConnectorNode connector, final String stepName, final StepConfiguration configuration) throws FlowUpdateException { + if (configurationProvider != null) { + final ConnectorWorkingConfiguration mergedConfiguration = buildMergedWorkingConfiguration(connector, stepName, configuration); + configurationProvider.save(connector.getIdentifier(), mergedConfiguration); + } + + connector.setConfiguration(stepName, configuration); + logger.info("Successfully configured {} for step {}", connector, stepName); + } + + @Override + public void inheritConfiguration(final ConnectorNode connector, final List activeFlowConfiguration, + final List workingFlowConfiguration, final Bundle flowContextBundle) throws FlowUpdateException { + + logger.debug("Inheriting configuration for {}", connector); + connector.transitionStateForUpdating(); + connector.prepareForUpdate(); + + try { + connector.inheritConfiguration(activeFlowConfiguration, workingFlowConfiguration, flowContextBundle); + logger.debug("Successfully inherited configuration for {}", connector); + } catch (final Exception e) { + logger.error("Failed to inherit configuration for {}", connector, e); + connector.abortUpdate(e); + connector.markInvalid("Flow Update Failure", "The flow could not be updated: " + e.getMessage()); + throw e; + } + } + + @Override + public void discardWorkingConfiguration(final ConnectorNode connector) { + if (configurationProvider != null) { + configurationProvider.discard(connector.getIdentifier()); + } + connector.discardWorkingConfiguration(); + cleanUpAssets(connector); + } + + @Override + public SecretsManager getSecretsManager() { + return secretsManager; + } + + @Override + public ConnectorStateTransition createStateTransition(final String type, final String id) { + final String componentDescription = "StandardConnectorNode[id=" + id + ", type=" + type + "]"; + return new StandardConnectorStateTransition(componentDescription); + } + + @Override + public FrameworkConnectorInitializationContextBuilder createInitializationContextBuilder() { + return new StandardConnectorInitializationContext.Builder(); + } + + @Override + public Asset storeAsset(final String connectorId, final String assetId, final String assetName, final InputStream content) throws IOException { + if (configurationProvider == null) { + return assetManager.saveAsset(connectorId, assetId, assetName, content); + } + + // When a provider is configured, delegate entirely to the provider, which should write to the AssetManager and sync to the external store. + try { + configurationProvider.storeAsset(connectorId, assetId, assetName, content); + } catch (final IOException e) { + throw e; + } catch (final Exception e) { + throw new IOException("Failed to store asset [" + assetName + "] to provider for connector [" + connectorId + "]", e); + } + logger.debug("Stored asset [nifiId={}, name={}] for connector [{}]", assetId, assetName, connectorId); + + return assetManager.getAsset(assetId) + .orElseThrow(() -> new IOException("Asset [" + assetId + "] not found after storing for connector [" + connectorId + "]")); + } + + @Override + public Optional getAsset(final String assetId) { + return assetManager.getAsset(assetId); + } + + @Override + public List getAssets(final String connectorId) { + return assetManager.getAssets(connectorId); + } + + @Override + public void deleteAssets(final String connectorId) { + final List assets = assetManager.getAssets(connectorId); + for (final Asset asset : assets) { + try { + if (configurationProvider != null) { + // When a provider is configured, delegate entirely to the provider, which should delete from the AssetManager and sync to the external store. + configurationProvider.deleteAsset(connectorId, asset.getIdentifier()); + } else { + assetManager.deleteAsset(asset.getIdentifier()); + } + } catch (final Exception e) { + logger.warn("Failed to delete asset [nifiUuid={}] for connector [{}]", asset.getIdentifier(), connectorId, e); + } + } + } + + @Override + public void syncAssetsFromProvider(final ConnectorNode connector) { + if (configurationProvider == null) { + return; + } + + final String connectorId = connector.getIdentifier(); + + // Step 1: Sync Connector Assets from Provider + logger.debug("Syncing assets from provider for connector [{}]", connectorId); + configurationProvider.syncAssets(connectorId); + + // Step 2: Sync ConnectorNode Configuration from Provider + syncFromProvider(connector); + } + + private void syncFromProvider(final ConnectorNode connector) { + if (configurationProvider == null) { + return; + } + + final String connectorId = connector.getIdentifier(); + final Optional externalConfig = configurationProvider.load(connectorId); + if (externalConfig.isEmpty()) { + return; + } + + final ConnectorWorkingConfiguration config = externalConfig.get(); + if (config.getName() != null) { + connector.setName(config.getName()); + } + + final List workingFlowConfiguration = config.getWorkingFlowConfiguration(); + if (workingFlowConfiguration != null) { + final MutableConnectorConfigurationContext workingConfigContext = connector.getWorkingFlowContext().getConfigurationContext(); + for (final VersionedConfigurationStep step : workingFlowConfiguration) { + final StepConfiguration stepConfiguration = toStepConfiguration(step); + workingConfigContext.replaceProperties(step.getName(), stepConfiguration); + } + } + } + + private ConnectorWorkingConfiguration buildWorkingConfiguration(final ConnectorNode connector) { + final ConnectorWorkingConfiguration config = new ConnectorWorkingConfiguration(); + config.setName(connector.getName()); + config.setWorkingFlowConfiguration(buildVersionedConfigurationSteps(connector.getWorkingFlowContext())); + return config; + } + + private List buildVersionedConfigurationSteps(final FrameworkFlowContext flowContext) { + if (flowContext == null) { + return List.of(); + } + + final ConnectorConfiguration configuration = flowContext.getConfigurationContext().toConnectorConfiguration(); + final List configurationSteps = new ArrayList<>(); + + for (final NamedStepConfiguration namedStepConfiguration : configuration.getNamedStepConfigurations()) { + final VersionedConfigurationStep versionedConfigurationStep = new VersionedConfigurationStep(); + versionedConfigurationStep.setName(namedStepConfiguration.stepName()); + versionedConfigurationStep.setProperties(toVersionedProperties(namedStepConfiguration.configuration())); + configurationSteps.add(versionedConfigurationStep); + } + + return configurationSteps; + } + + private ConnectorWorkingConfiguration buildMergedWorkingConfiguration(final ConnectorNode connector, final String stepName, final StepConfiguration incomingConfiguration) { + final ConnectorWorkingConfiguration existingConfig; + if (configurationProvider != null) { + final Optional externalConfig = configurationProvider.load(connector.getIdentifier()); + existingConfig = externalConfig.orElseGet(() -> buildWorkingConfiguration(connector)); + } else { + existingConfig = buildWorkingConfiguration(connector); + } + + final List existingSteps = existingConfig.getWorkingFlowConfiguration() != null + ? new ArrayList<>(existingConfig.getWorkingFlowConfiguration()) + : new ArrayList<>(); + + VersionedConfigurationStep targetStep = null; + for (final VersionedConfigurationStep step : existingSteps) { + if (stepName.equals(step.getName())) { + targetStep = step; + break; + } + } + + if (targetStep == null) { + targetStep = new VersionedConfigurationStep(); + targetStep.setName(stepName); + targetStep.setProperties(new HashMap<>()); + existingSteps.add(targetStep); + } + + final Map mergedProperties = targetStep.getProperties() != null + ? new HashMap<>(targetStep.getProperties()) + : new HashMap<>(); + + for (final Map.Entry entry : incomingConfiguration.getPropertyValues().entrySet()) { + if (entry.getValue() != null) { + mergedProperties.put(entry.getKey(), toVersionedValueReference(entry.getValue())); + } + } + targetStep.setProperties(mergedProperties); + + existingConfig.setWorkingFlowConfiguration(existingSteps); + return existingConfig; + } + + private Map toVersionedProperties(final StepConfiguration configuration) { + final Map versionedProperties = new HashMap<>(); + for (final Map.Entry entry : configuration.getPropertyValues().entrySet()) { + final ConnectorValueReference valueReference = entry.getValue(); + if (valueReference != null) { + versionedProperties.put(entry.getKey(), toVersionedValueReference(valueReference)); + } + } + return versionedProperties; + } + + private VersionedConnectorValueReference toVersionedValueReference(final ConnectorValueReference valueReference) { + final VersionedConnectorValueReference versionedReference = new VersionedConnectorValueReference(); + versionedReference.setValueType(valueReference.getValueType().name()); + + switch (valueReference) { + case StringLiteralValue stringLiteral -> versionedReference.setValue(stringLiteral.getValue()); + case AssetReference assetRef -> versionedReference.setAssetIds(assetRef.getAssetIdentifiers()); + case SecretReference secretRef -> { + versionedReference.setProviderId(secretRef.getProviderId()); + versionedReference.setProviderName(secretRef.getProviderName()); + versionedReference.setSecretName(secretRef.getSecretName()); + versionedReference.setFullyQualifiedSecretName(secretRef.getFullyQualifiedName()); + } + } + + return versionedReference; + } + + private StepConfiguration toStepConfiguration(final VersionedConfigurationStep step) { + final Map propertyValues = new HashMap<>(); + final Map versionedProperties = step.getProperties(); + if (versionedProperties != null) { + for (final Map.Entry entry : versionedProperties.entrySet()) { + final VersionedConnectorValueReference versionedRef = entry.getValue(); + if (versionedRef != null) { + propertyValues.put(entry.getKey(), toConnectorValueReference(versionedRef)); + } + } + } + return new StepConfiguration(propertyValues); + } + + private ConnectorValueReference toConnectorValueReference(final VersionedConnectorValueReference versionedReference) { + final ConnectorValueType valueType = ConnectorValueType.valueOf(versionedReference.getValueType()); + return switch (valueType) { + case STRING_LITERAL -> new StringLiteralValue(versionedReference.getValue()); + case ASSET_REFERENCE -> new AssetReference(versionedReference.getAssetIds()); + case SECRET_REFERENCE -> new SecretReference(versionedReference.getProviderId(), versionedReference.getProviderName(), + versionedReference.getSecretName(), versionedReference.getFullyQualifiedSecretName()); + }; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorStateTransition.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorStateTransition.java new file mode 100644 index 000000000000..9cfd4cccb382 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorStateTransition.java @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.atomic.AtomicReference; + +public class StandardConnectorStateTransition implements ConnectorStateTransition { + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorStateTransition.class); + + private final String componentDescription; + private final AtomicReference currentState = new AtomicReference<>(ConnectorState.STOPPED); + private final AtomicReference desiredState = new AtomicReference<>(ConnectorState.STOPPED); + private final List> pendingStartFutures = new ArrayList<>(); + private final List> pendingStopFutures = new ArrayList<>(); + + public StandardConnectorStateTransition(final String componentDescription) { + this.componentDescription = componentDescription; + } + + @Override + public ConnectorState getCurrentState() { + return currentState.get(); + } + + @Override + public ConnectorState getDesiredState() { + return desiredState.get(); + } + + @Override + public void setDesiredState(final ConnectorState desiredState) { + this.desiredState.set(desiredState); + logger.info("Desired State for {} set to {}", componentDescription, desiredState); + } + + @Override + public boolean trySetCurrentState(final ConnectorState expectedState, final ConnectorState newState) { + final boolean changed = currentState.compareAndSet(expectedState, newState); + if (changed) { + logger.info("Transitioned current state for {} from {} to {}", componentDescription, expectedState, newState); + completeFuturesForStateTransition(newState); + } + + return changed; + } + + @Override + public void setCurrentState(final ConnectorState newState) { + final ConnectorState oldState = currentState.getAndSet(newState); + logger.info("Transitioned current state for {} from {} to {}", componentDescription, oldState, newState); + completeFuturesForStateTransition(newState); + } + + @Override + public synchronized void addPendingStartFuture(final CompletableFuture future) { + pendingStartFutures.add(future); + } + + @Override + public synchronized void addPendingStopFuture(final CompletableFuture future) { + pendingStopFutures.add(future); + } + + private synchronized void completeFuturesForStateTransition(final ConnectorState newState) { + if (newState == ConnectorState.RUNNING) { + final List> futuresToComplete = new ArrayList<>(pendingStartFutures); + pendingStartFutures.clear(); + + for (final CompletableFuture future : futuresToComplete) { + future.complete(null); + } + + if (!futuresToComplete.isEmpty()) { + logger.debug("Completed {} pending start futures for {}", futuresToComplete.size(), componentDescription); + } + } + + if (newState == ConnectorState.STOPPED) { + final List> futuresToComplete = new ArrayList<>(pendingStopFutures); + pendingStopFutures.clear(); + + for (final CompletableFuture future : futuresToComplete) { + future.complete(null); + } + + if (!futuresToComplete.isEmpty()) { + logger.debug("Completed {} pending stop futures for {}", futuresToComplete.size(), componentDescription); + } + } + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorValidationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorValidationContext.java new file mode 100644 index 000000000000..61d36dfe8c13 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorValidationContext.java @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterLookup; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +public class StandardConnectorValidationContext implements ConnectorValidationContext { + private final DescribedValueProvider allowableValueProvider; + private final Map stepConfigurations; + private final ParameterLookup parameterLookup; + + public StandardConnectorValidationContext(final ConnectorConfiguration connectorConfiguration, final DescribedValueProvider allowableValueProvider, + final ParameterContextFacade parameterContextFacade) { + + this.allowableValueProvider = allowableValueProvider; + + stepConfigurations = new HashMap<>(); + for (final NamedStepConfiguration stepConfiguration : connectorConfiguration.getNamedStepConfigurations()) { + stepConfigurations.put(stepConfiguration.stepName(), stepConfiguration.configuration()); + } + + parameterLookup = new ParameterLookupBridge(parameterContextFacade); + } + + @Override + public ValidationContext createValidationContext(final String stepName, final String propertyGroupName) { + final StepConfiguration stepConfig = stepConfigurations.get(stepName); + final Map stepProperties = stepConfig != null ? stepConfig.getPropertyValues() : Collections.emptyMap(); + + final Map stringValues = new HashMap<>(); + for (final Map.Entry entry : stepProperties.entrySet()) { + final ConnectorValueReference valueRef = entry.getValue(); + + if (valueRef instanceof StringLiteralValue stringLiteral) { + stringValues.put(entry.getKey(), stringLiteral.getValue()); + } else { + stringValues.put(entry.getKey(), null); + } + } + + return new ConnectorValidationContextBridge(stringValues, parameterLookup); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName) { + return allowableValueProvider.fetchAllowableValues(stepName, propertyName); + } + + private static class ParameterLookupBridge implements ParameterLookup { + private final ParameterContextFacade parameterContextFacade; + + public ParameterLookupBridge(final ParameterContextFacade parameterContextFacade) { + this.parameterContextFacade = parameterContextFacade; + } + + @Override + public Optional getParameter(final String parameterName) { + final String parameterValue = parameterContextFacade.getValue(parameterName); + if (parameterValue == null) { + return Optional.empty(); + } + + final Parameter parameter = new Parameter.Builder() + .name(parameterName) + .provided(false) + .sensitive(parameterContextFacade.isSensitive(parameterName)) + .value(parameterValue) + .build(); + return Optional.of(parameter); + } + + @Override + public boolean isEmpty() { + return parameterContextFacade.getDefinedParameterNames().isEmpty(); + } + + @Override + public long getVersion() { + return 0; + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardFlowContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardFlowContext.java new file mode 100644 index 000000000000..3d56a999af81 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardFlowContext.java @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.components.FlowContextType; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.parameter.ParameterContext; + +import java.util.HashMap; +import java.util.Map; + +public class StandardFlowContext implements FrameworkFlowContext { + private final ProcessGroup managedProcessGroup; + private final MutableConnectorConfigurationContext configurationContext; + private final ProcessGroupFacadeFactory groupFacadeFactory; + private final ParameterContextFacadeFactory parameterContextFacadeFactory; + private final ComponentLog connectorLog; + private final FlowContextType flowContextType; + private final Bundle bundle; + + private volatile ProcessGroupFacade rootGroup; + private volatile ParameterContextFacade parameterContext; + + + public StandardFlowContext(final ProcessGroup managedProcessGroup, final MutableConnectorConfigurationContext configurationContext, + final ProcessGroupFacadeFactory groupFacadeFactory, final ParameterContextFacadeFactory parameterContextFacadeFactory, + final ComponentLog connectorLog, final FlowContextType flowContextType, + final Bundle bundle) { + + this.managedProcessGroup = managedProcessGroup; + this.groupFacadeFactory = groupFacadeFactory; + this.parameterContextFacadeFactory = parameterContextFacadeFactory; + this.connectorLog = connectorLog; + this.configurationContext = configurationContext; + this.flowContextType = flowContextType; + this.bundle = bundle; + + this.rootGroup = groupFacadeFactory.create(managedProcessGroup, connectorLog); + this.parameterContext = parameterContextFacadeFactory.create(managedProcessGroup); + } + + @Override + public ProcessGroupFacade getRootGroup() { + return rootGroup; + } + + @Override + public ParameterContextFacade getParameterContext() { + return parameterContext; + } + + @Override + public MutableConnectorConfigurationContext getConfigurationContext() { + return configurationContext; + } + + @Override + public void updateFlow(final VersionedExternalFlow versionedExternalFlow, final AssetManager assetManager) throws FlowUpdateException { + if (versionedExternalFlow.getParameterContexts() == null) { + versionedExternalFlow.setParameterContexts(new HashMap<>()); + } + + try { + managedProcessGroup.verifyCanUpdate(versionedExternalFlow, true, false); + } catch (final IllegalStateException e) { + throw new FlowUpdateException("Flow is not in a state that allows the requested updated", e); + } + + final ParameterContext managedGroupParameterContext = managedProcessGroup.getParameterContext(); + updateParameterContextNames(versionedExternalFlow.getFlowContents(), managedGroupParameterContext.getName()); + + final VersionedExternalFlow externalFlowWithResolvedParameters = new VersionedExternalFlow(); + externalFlowWithResolvedParameters.setFlowContents(versionedExternalFlow.getFlowContents()); + externalFlowWithResolvedParameters.setParameterContexts(Map.of()); + + managedProcessGroup.updateFlow(externalFlowWithResolvedParameters, managedProcessGroup.getIdentifier(), false, true, true); + + rootGroup = groupFacadeFactory.create(managedProcessGroup, connectorLog); + + final ConnectorParameterLookup parameterLookup = new ConnectorParameterLookup(versionedExternalFlow.getParameterContexts().values(), assetManager); + getParameterContext().updateParameters(parameterLookup.getParameterValues()); + parameterContext = parameterContextFacadeFactory.create(managedProcessGroup); + } + + private void updateParameterContextNames(final VersionedProcessGroup group, final String parameterContextName) { + group.setParameterContextName(parameterContextName); + if (group.getProcessGroups() != null) { + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + updateParameterContextNames(childGroup, parameterContextName); + } + } + } + + @Override + public Bundle getBundle() { + return bundle; + } + + @Override + public FlowContextType getType() { + return flowContextType; + } + + @Override + public ProcessGroup getManagedProcessGroup() { + return managedProcessGroup; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardStepConfigurationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardStepConfigurationContext.java new file mode 100644 index 000000000000..b728033d899d --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardStepConfigurationContext.java @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.util.HashMap; +import java.util.Map; +import java.util.Set; + +public class StandardStepConfigurationContext implements StepConfigurationContext { + private final String stepName; + private final ConnectorConfigurationContext parentContext; + + public StandardStepConfigurationContext(final String stepName, final ConnectorConfigurationContext parentContext) { + this.stepName = stepName; + this.parentContext = parentContext; + } + + @Override + public ConnectorPropertyValue getProperty(final String propertyName) { + return parentContext.getProperty(stepName, propertyName); + } + + @Override + public ConnectorPropertyValue getProperty(final ConnectorPropertyDescriptor propertyDescriptor) { + return parentContext.getProperty(stepName, propertyDescriptor.getName()); + } + + @Override + public StepConfigurationContext createWithOverrides(final Map propertyValues) { + final ConnectorConfigurationContext updatedContext = parentContext.createWithOverrides(stepName, propertyValues); + return updatedContext.scopedToStep(stepName); + } + + @Override + public Map getProperties() { + final Map properties = new HashMap<>(); + final Set propertyNames = parentContext.getPropertyNames(stepName); + for (final String propertyName : propertyNames) { + properties.put(propertyName, getProperty(propertyName)); + } + + return properties; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/ComponentContextProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/ComponentContextProvider.java new file mode 100644 index 000000000000..ed03fc21b4aa --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/ComponentContextProvider.java @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.parameter.ParameterLookup; +import org.apache.nifi.processor.ProcessContext; + +import java.util.Map; + +public interface ComponentContextProvider { + ProcessContext createProcessContext(ProcessorNode processorNode, ParameterLookup parameterLookup); + + ProcessContext createProcessContext(ProcessorNode processorNode, Map propertiesOverride, ParameterLookup parameterLookup); + + ValidationContext createValidationContext(ProcessorNode processorNode, Map properties, ParameterLookup parameterLookup); + + ConfigurationContext createConfigurationContext(ControllerServiceNode serviceNode, ParameterLookup parameterLookup); + + ConfigurationContext createConfigurationContext(ControllerServiceNode serviceNode, Map propertiesOverride, ParameterLookup parameterLookup); + + ValidationContext createValidationContext(ControllerServiceNode serviceNode, Map properties, ParameterLookup parameterLookup); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/IllegalExecutionEngineStatelessGroupLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/IllegalExecutionEngineStatelessGroupLifecycle.java new file mode 100644 index 000000000000..79e25e0d1a93 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/IllegalExecutionEngineStatelessGroupLifecycle.java @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; +import org.apache.nifi.groups.ProcessGroup; + +import java.util.concurrent.CompletableFuture; + +public class IllegalExecutionEngineStatelessGroupLifecycle implements StatelessGroupLifecycle { + private final ProcessGroup processGroup; + + public IllegalExecutionEngineStatelessGroupLifecycle(final ProcessGroup processGroup) { + this.processGroup = processGroup; + } + + @Override + public CompletableFuture start() { + throw new IllegalStateException("Cannot start " + processGroup + " as a Stateless Group because the Process Group is not configured to run using the Stateless Execution Engine"); + } + + @Override + public CompletableFuture stop() { + throw new IllegalStateException("Cannot stop " + processGroup + " as a Stateless Group because the Process Group is not configured to run using the Stateless Execution Engine"); + } + + @Override + public CompletableFuture terminate() { + throw new IllegalStateException("Cannot terminate " + processGroup + " as a Stateless Group because the Process Group is not configured to run using the Stateless Execution Engine"); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneConnectionFacade.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneConnectionFacade.java new file mode 100644 index 000000000000..cc21353731d5 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneConnectionFacade.java @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.connector.DropFlowFileSummary; +import org.apache.nifi.components.connector.components.ConnectionFacade; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flowfile.FlowFile; + +import java.io.IOException; +import java.util.Objects; +import java.util.function.Predicate; + +public class StandaloneConnectionFacade implements ConnectionFacade { + private final Connection connection; + private final VersionedConnection versionedConnection; + + public StandaloneConnectionFacade(final Connection connection, final VersionedConnection versionedConnection) { + this.connection = connection; + this.versionedConnection = versionedConnection; + } + + @Override + public VersionedConnection getDefinition() { + return versionedConnection; + } + + @Override + public QueueSize getQueueSize() { + return connection.getFlowFileQueue().size(); + } + + @Override + public void purge() { + // TODO: Require arguments here + connection.getFlowFileQueue().dropFlowFiles("User requested purge", "User"); + } + + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + return connection.getFlowFileQueue().dropFlowFiles(predicate); + } + + @Override + public String toString() { + return "StandaloneConnectionFacade[id=" + versionedConnection.getIdentifier() + ", name=" + versionedConnection.getName() + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final StandaloneConnectionFacade that = (StandaloneConnectionFacade) o; + return Objects.equals(versionedConnection.getIdentifier(), that.versionedConnection.getIdentifier()); + } + + @Override + public int hashCode() { + return Objects.hashCode(versionedConnection.getIdentifier()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneControllerServiceFacade.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneControllerServiceFacade.java new file mode 100644 index 000000000000..808b5a4747dc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneControllerServiceFacade.java @@ -0,0 +1,243 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.ConnectorParameterLookup; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ControllerServiceFacade; +import org.apache.nifi.components.connector.components.ControllerServiceLifecycle; +import org.apache.nifi.components.validation.DisabledServiceValidationResult; +import org.apache.nifi.components.validation.ValidationState; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterLookup; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +public class StandaloneControllerServiceFacade implements ControllerServiceFacade { + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) + .setDefaultPropertyInclusion(JsonInclude.Include.NON_NULL); + + private final ControllerServiceNode controllerServiceNode; + private final VersionedControllerService versionedControllerService; + private final ParameterContext parameterContext; + private final ControllerServiceLifecycle lifecycle; + private final ComponentContextProvider componentContextProvider; + private final ComponentLog connectorLogger; + private final ExtensionManager extensionManager; + private final AssetManager assetManager; + + public StandaloneControllerServiceFacade(final ControllerServiceNode controllerServiceNode, final VersionedControllerService versionedControllerService, + final ParameterContext parameterContext, final ProcessScheduler processScheduler, final ComponentContextProvider componentContextProvider, final ComponentLog connectorLogger, + final ExtensionManager extensionManager, final AssetManager assetManager) { + + this.controllerServiceNode = controllerServiceNode; + this.versionedControllerService = versionedControllerService; + this.parameterContext = parameterContext; + this.componentContextProvider = componentContextProvider; + this.connectorLogger = connectorLogger; + this.extensionManager = extensionManager; + this.assetManager = assetManager; + + this.lifecycle = new StandaloneControllerServiceLifecycle(controllerServiceNode, processScheduler, componentContextProvider, parameterContext); + } + + @Override + public VersionedControllerService getDefinition() { + return versionedControllerService; + } + + @Override + public ControllerServiceLifecycle getLifecycle() { + return lifecycle; + } + + @Override + public List validate() { + return validate(getDefinition().getProperties()); + } + + // TODO: Refactor to avoid duplicate code with StandaloneProcessorFacade + @Override + public List validate(final Map propertyValues) { + final ValidationContext validationContext = componentContextProvider.createValidationContext(controllerServiceNode, propertyValues, parameterContext); + final ValidationState validationState = controllerServiceNode.performValidation(validationContext); + final List validationErrors = validationState.getValidationErrors().stream() + .filter(result -> !result.isValid()) + .filter(result -> !DisabledServiceValidationResult.isMatch(result)) + .toList(); + + return validationErrors; + } + + @Override + public List verify(final Map propertyValues, final Map variables) { + final ConfigurationContext configurationContext = componentContextProvider.createConfigurationContext(controllerServiceNode, propertyValues, parameterContext); + return controllerServiceNode.verifyConfiguration(configurationContext, connectorLogger, variables, extensionManager, parameterContext); + } + + @Override + public List verify(final Map propertyValues, final VersionedParameterContext parameterContext, final Map variables) { + final ParameterLookup parameterLookup = new ConnectorParameterLookup(List.of(parameterContext), assetManager); + final ConfigurationContext configurationContext = componentContextProvider.createConfigurationContext(controllerServiceNode, propertyValues, parameterLookup); + return controllerServiceNode.verifyConfiguration(configurationContext, connectorLogger, variables, extensionManager, parameterLookup); + } + + @Override + public List verify(final VersionedExternalFlow versionedExternalFlow, final Map variables) { + final VersionedProcessGroup parentGroup = findVersionedServiceGroup(versionedExternalFlow.getFlowContents()); + final VersionedControllerService versionedService = findVersionedService(parentGroup); + if (versionedService == null) { + throw new IllegalArgumentException("Cannot verify Controller Service with ID " + versionedControllerService.getIdentifier() + " because it does not exist in the provided flow"); + } + + final ConnectorParameterLookup connectorParameterLookup = new ConnectorParameterLookup(versionedExternalFlow.getParameterContexts().values(), assetManager); + + final Map propertyValues = versionedService.getProperties(); + final ConfigurationContext configurationContext = componentContextProvider.createConfigurationContext(controllerServiceNode, propertyValues, connectorParameterLookup); + return controllerServiceNode.verifyConfiguration(configurationContext, connectorLogger, variables, extensionManager, connectorParameterLookup); + } + + private VersionedProcessGroup findVersionedServiceGroup(final VersionedProcessGroup group) { + for (final VersionedControllerService service : group.getControllerServices()) { + if (service.getIdentifier().equals(versionedControllerService.getIdentifier())) { + return group; + } + } + + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + final VersionedProcessGroup serviceGroup = findVersionedServiceGroup(childGroup); + if (serviceGroup != null) { + return serviceGroup; + } + } + + return null; + } + + private VersionedControllerService findVersionedService(final VersionedProcessGroup group) { + if (group == null) { + return null; + } + + final VersionedProcessGroup parentGroup = findVersionedServiceGroup(group); + if (parentGroup == null) { + return null; + } + + for (final VersionedControllerService service : parentGroup.getControllerServices()) { + if (service.getIdentifier().equals(controllerServiceNode.getVersionedComponentId().orElseThrow())) { + return service; + } + } + + return null; + } + + @Override + public Object invokeConnectorMethod(final String methodName, final Map arguments) throws InvocationFailedException { + final Map jsonArguments = serializeArgumentsToJson(arguments); + final ConfigurationContext configurationContext = componentContextProvider.createConfigurationContext(controllerServiceNode, parameterContext); + final String jsonResult = controllerServiceNode.invokeConnectorMethod(methodName, jsonArguments, configurationContext); + if (jsonResult == null) { + return null; + } + + try { + return OBJECT_MAPPER.readValue(jsonResult, Object.class); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to deserialize return value from Connector Method '" + methodName + "'", e); + } + } + + @Override + public T invokeConnectorMethod(final String methodName, final Map arguments, final Class returnType) throws InvocationFailedException { + final Map jsonArguments = serializeArgumentsToJson(arguments); + final ConfigurationContext configurationContext = componentContextProvider.createConfigurationContext(controllerServiceNode, parameterContext); + final String jsonResult = controllerServiceNode.invokeConnectorMethod(methodName, jsonArguments, configurationContext); + if (jsonResult == null) { + return null; + } + + try { + return OBJECT_MAPPER.readValue(jsonResult, returnType); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to deserialize return value from Connector Method '" + methodName + "'", e); + } + } + + @Override + public String toString() { + final String type = versionedControllerService.getType(); + final String simpleType = type == null ? null : type.substring(type.lastIndexOf('.') + 1); + return "StandaloneControllerServiceFacade[id=" + versionedControllerService.getIdentifier() + ", name=" + versionedControllerService.getName() + ", type=" + simpleType + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final StandaloneControllerServiceFacade that = (StandaloneControllerServiceFacade) o; + return Objects.equals(versionedControllerService.getIdentifier(), that.versionedControllerService.getIdentifier()); + } + + @Override + public int hashCode() { + return Objects.hashCode(versionedControllerService.getIdentifier()); + } + + private Map serializeArgumentsToJson(final Map arguments) throws InvocationFailedException { + final Map jsonArguments = new HashMap<>(); + for (final Map.Entry entry : arguments.entrySet()) { + if (entry.getValue() == null) { + jsonArguments.put(entry.getKey(), null); + } else { + try { + jsonArguments.put(entry.getKey(), OBJECT_MAPPER.writeValueAsString(entry.getValue())); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to serialize argument '" + entry.getKey() + "' to JSON", e); + } + } + } + return jsonArguments; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneControllerServiceLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneControllerServiceLifecycle.java new file mode 100644 index 000000000000..c8016ce67a4d --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneControllerServiceLifecycle.java @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.connector.components.ControllerServiceLifecycle; +import org.apache.nifi.components.connector.components.ControllerServiceState; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.parameter.ParameterContext; + +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +public class StandaloneControllerServiceLifecycle implements ControllerServiceLifecycle { + private final ControllerServiceNode controllerServiceNode; + private final ProcessScheduler processScheduler; + private final ComponentContextProvider componentContextProvider; + private final ParameterContext parameterContext; + + public StandaloneControllerServiceLifecycle(final ControllerServiceNode controllerServiceNode, final ProcessScheduler scheduler, + final ComponentContextProvider componentContextProvider, final ParameterContext parameterContext) { + + this.controllerServiceNode = controllerServiceNode; + this.processScheduler = scheduler; + this.componentContextProvider = componentContextProvider; + this.parameterContext = parameterContext; + } + + @Override + public ControllerServiceState getState() { + return switch (controllerServiceNode.getState()) { + case DISABLED -> ControllerServiceState.DISABLED; + case ENABLING -> ControllerServiceState.ENABLING; + case ENABLED -> ControllerServiceState.ENABLED; + case DISABLING -> ControllerServiceState.DISABLING; + }; + } + + @Override + public CompletableFuture enable() { + // If validating, perform validation to ensure it's complete before enabling + final ValidationStatus currentStatus = controllerServiceNode.getValidationStatus(); + if (currentStatus != ValidationStatus.VALID) { + controllerServiceNode.performValidation(); + } + + return processScheduler.enableControllerService(controllerServiceNode); + } + + @Override + public CompletableFuture enable(final Map propertyValueOverrides) { + final ConfigurationContext configurationContext = componentContextProvider.createConfigurationContext(controllerServiceNode, propertyValueOverrides, parameterContext); + return processScheduler.enableControllerService(controllerServiceNode, configurationContext); + } + + @Override + public CompletableFuture disable() { + return processScheduler.disableControllerService(controllerServiceNode); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneParameterContextFacade.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneParameterContextFacade.java new file mode 100644 index 000000000000..2599269d3554 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneParameterContextFacade.java @@ -0,0 +1,229 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.connector.components.ParameterValue; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.serialization.AffectedComponentSet; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterDescriptor; +import org.apache.nifi.parameter.ParameterReferenceManager; +import org.apache.nifi.parameter.StandardParameterReferenceManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.InputStream; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; + +public class StandaloneParameterContextFacade implements ParameterContextFacade { + private static final Logger logger = LoggerFactory.getLogger(StandaloneParameterContextFacade.class); + + private final FlowController flowController; + private final ProcessGroup managedProcessGroup; + private final ParameterContext parameterContext; + + public StandaloneParameterContextFacade(final FlowController flowController, final ProcessGroup managedProcessGroup) { + this.flowController = flowController; + this.managedProcessGroup = managedProcessGroup; + this.parameterContext = managedProcessGroup.getParameterContext(); + } + + @Override + public void updateParameters(final Collection updatedValues) { + // Check if the parameter already exists and if the sensitivity matches + validateParameterSensitivities(updatedValues); + + final AffectedComponentSet affectedComponentSet = new AffectedComponentSet(flowController); + final ParameterReferenceManager parameterReferenceManager = new StandardParameterReferenceManager(() -> managedProcessGroup); + + final Set allReferencingProcessors = new HashSet<>(); + final Set allReferencingServices = new HashSet<>(); + for (final ParameterValue parameterValue : updatedValues) { + if (!isParameterChanged(parameterValue)) { + continue; + } + + final String parameterName = parameterValue.getName(); + final Set referencingProcessors = parameterReferenceManager.getProcessorsReferencing(managedProcessGroup.getParameterContext(), parameterName); + final Set referencingServices = parameterReferenceManager.getControllerServicesReferencing(managedProcessGroup.getParameterContext(), parameterName); + referencingProcessors.forEach(affectedComponentSet::addProcessor); + referencingServices.forEach(affectedComponentSet::addControllerService); + + allReferencingProcessors.addAll(referencingProcessors); + allReferencingServices.addAll(referencingServices); + } + + if (affectedComponentSet.isEmpty()) { + logger.info("No components are affected by the parameter update; no need to restart anything."); + return; + } + + final AffectedComponentSet activeSet = affectedComponentSet.toActiveSet(); + logger.info("Stopping {} affected components due to parameter update", activeSet.getComponentCount()); + activeSet.stop(); + + // Update the parameter value in our map + final Map updatedParameters = createParameterMap(updatedValues); + managedProcessGroup.getParameterContext().setParameters(updatedParameters); + + allReferencingProcessors.forEach(ProcessorNode::resetValidationState); + allReferencingServices.forEach(ControllerServiceNode::resetValidationState); + + logger.info("Parameter Context updated {} parameter. Restarting {} affected components.", updatedValues.size(), activeSet.getComponentCount()); + activeSet.start(); + } + + private Map createParameterMap(final Collection parameterValues) { + final Map paramMap = new HashMap<>(); + for (final ParameterValue parameterValue : parameterValues) { + final Parameter parameter = new Parameter.Builder() + .name(parameterValue.getName()) + .value(parameterValue.getValue()) + .sensitive(parameterValue.isSensitive()) + .build(); + + paramMap.put(parameterValue.getName(), parameter); + } + + return paramMap; + } + + private boolean isParameterChanged(final ParameterValue parameterValue) { + final String parameterName = parameterValue.getName(); + final Optional optionalParameter = parameterContext.getParameter(parameterName); + if (optionalParameter.isEmpty()) { + return true; // New parameter + } + + final Parameter existingParameter = optionalParameter.get(); + if (!Objects.equals(existingParameter.getValue(), parameterValue.getValue())) { + return true; // Value changed + } + + if (!Objects.equals(existingParameter.getReferencedAssets(), parameterValue.getAssets())) { + return true; // Assets changed + } + + return false; + } + + private void validateParameterSensitivities(final Collection updatedValues) { + for (final ParameterValue parameterValue : updatedValues) { + final String parameterName = parameterValue.getName(); + final boolean sensitive = parameterValue.isSensitive(); + + final Optional optionalParameter = parameterContext.getParameter(parameterName); + if (optionalParameter.isEmpty()) { + continue; // New parameter, no existing sensitivity to check + } + + final boolean existingSensitive = optionalParameter.get().getDescriptor().isSensitive(); + if (existingSensitive != sensitive) { + throw new IllegalStateException("Cannot update parameter " + parameterName + " because the Parameter is already defined as being " + + (existingSensitive ? "sensitive" : "non-sensitive") + " but the update is defined as being " + + (sensitive ? "sensitive" : "non-sensitive")); + } + } + } + + @Override + public synchronized String getValue(final String parameterName) { + final Optional optionalParameter = parameterContext.getParameter(parameterName); + if (optionalParameter.isEmpty()) { + return null; + } + + final Parameter existing = optionalParameter.get(); + final String explicitValue = existing.getValue(); + if (explicitValue != null) { + return explicitValue; + } + + final List assets = existing.getReferencedAssets(); + if (assets.isEmpty()) { + return null; + } + + final StringBuilder sb = new StringBuilder(); + for (final Asset asset : assets) { + sb.append(asset.getFile().getAbsolutePath()); + sb.append(","); + } + sb.setLength(sb.length() - 1); // Remove trailing comma + return sb.toString(); + } + + @Override + public synchronized Set getDefinedParameterNames() { + final Set parameterNames = new HashSet<>(); + for (final ParameterDescriptor descriptor : parameterContext.getEffectiveParameters().keySet()) { + parameterNames.add(descriptor.getName()); + } + return parameterNames; + } + + @Override + public synchronized boolean isSensitive(final String parameterName) { + return parameterContext.getParameter(parameterName) + .map(Parameter::getDescriptor) + .map(ParameterDescriptor::isSensitive) + .orElse(false); + } + + @Override + public Asset createAsset(final InputStream inputStream) throws IOException { + // TODO: Implement + return null; + } + + @Override + public String toString() { + return "StandaloneParameterContextFacade[id=" + parameterContext.getIdentifier() + ", name=" + parameterContext.getName() + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final StandaloneParameterContextFacade that = (StandaloneParameterContextFacade) o; + return Objects.equals(parameterContext.getIdentifier(), that.parameterContext.getIdentifier()); + } + + @Override + public int hashCode() { + return Objects.hashCode(parameterContext.getIdentifier()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessGroupFacade.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessGroupFacade.java new file mode 100644 index 000000000000..83cd0a471917 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessGroupFacade.java @@ -0,0 +1,424 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.DropFlowFileSummary; +import org.apache.nifi.components.connector.components.ComponentHierarchyScope; +import org.apache.nifi.components.connector.components.ConnectionFacade; +import org.apache.nifi.components.connector.components.ControllerServiceFacade; +import org.apache.nifi.components.connector.components.ControllerServiceReferenceScope; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.components.connector.components.ProcessGroupLifecycle; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.controller.service.ControllerServiceProvider; +import org.apache.nifi.flow.ExecutionEngine; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.parameter.ParameterContext; + +import java.io.IOException; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; +import java.util.function.Predicate; +import java.util.stream.Collectors; + +/** + * An implementation of ProcessGroupFacade that implements the functionality for a standalone + * (non-clustered) NiFi instance. + */ +public class StandaloneProcessGroupFacade implements ProcessGroupFacade { + private final ProcessGroup processGroup; + private final VersionedProcessGroup flowDefinition; + private final ProcessScheduler processScheduler; + private final ParameterContext parameterContext; + private final Map processorMap; + private final Map controllerServiceMap; + private final Map connectionMap; + private final Map processGroupMap; + private final ControllerServiceProvider controllerServiceProvider; + private final StandaloneProcessGroupLifecycle lifecycle; + private final StatelessGroupLifecycle statelessGroupLifecycle; + private final ComponentContextProvider componentContextProvider; + private final ComponentLog connectorLogger; + private final ExtensionManager extensionManager; + private final AssetManager assetManager; + + public StandaloneProcessGroupFacade(final ProcessGroup processGroup, final VersionedProcessGroup flowDefinition, final ProcessScheduler processScheduler, + final ParameterContext parameterContext, final ControllerServiceProvider controllerServiceProvider, final ComponentContextProvider componentContextProvider, + final ComponentLog connectorLogger, final ExtensionManager extensionManager, final AssetManager assetManager) { + this.processGroup = processGroup; + this.flowDefinition = flowDefinition; + this.processScheduler = processScheduler; + this.parameterContext = parameterContext; + this.controllerServiceProvider = controllerServiceProvider; + this.componentContextProvider = componentContextProvider; + this.connectorLogger = connectorLogger; + this.extensionManager = extensionManager; + this.assetManager = assetManager; + + this.processorMap = mapProcessors(flowDefinition); + this.controllerServiceMap = mapControllerServices(flowDefinition); + this.connectionMap = mapConnections(flowDefinition); + this.processGroupMap = mapProcessGroups(flowDefinition); + + final ExecutionEngine executionEngine = processGroup.resolveExecutionEngine(); + if (executionEngine == ExecutionEngine.STATELESS) { + this.statelessGroupLifecycle = new StandaloneStatelessGroupLifecycle(processGroup, processScheduler); + } else { + this.statelessGroupLifecycle = new IllegalExecutionEngineStatelessGroupLifecycle(processGroup); + } + + this.lifecycle = new StandaloneProcessGroupLifecycle(processGroup, controllerServiceProvider, statelessGroupLifecycle, + id -> getProcessGroup(id).getLifecycle()); + } + + private Map mapProcessors(final VersionedProcessGroup flowDefinition) { + final Map processors = new HashMap<>(); + flowDefinition.getProcessors().forEach(processor -> processors.put(processor.getIdentifier(), processor)); + return processors; + } + + private Map mapControllerServices(final VersionedProcessGroup flowDefinition) { + final Map controllerServices = new HashMap<>(); + flowDefinition.getControllerServices().forEach(controllerService -> controllerServices.put(controllerService.getIdentifier(), controllerService)); + return controllerServices; + } + + private Map mapConnections(final VersionedProcessGroup flowDefinition) { + final Map connections = new HashMap<>(); + flowDefinition.getConnections().forEach(connection -> connections.put(connection.getIdentifier(), connection)); + return connections; + } + + private Map mapProcessGroups(final VersionedProcessGroup flowDefinition) { + final Map processGroups = new HashMap<>(); + flowDefinition.getProcessGroups().forEach(processGroup -> processGroups.put(processGroup.getIdentifier(), processGroup)); + return processGroups; + } + + @Override + public VersionedProcessGroup getDefinition() { + return flowDefinition; + } + + @Override + public ProcessorFacade getProcessor(final String id) { + final ProcessorNode processorNode = lookupProcessorNode(id); + if (processorNode == null) { + return null; + } + + final VersionedProcessor processor = processorMap.get(id); + if (processor == null) { + return null; + } + + return new StandaloneProcessorFacade(processorNode, processor, processScheduler, parameterContext, + componentContextProvider, connectorLogger, extensionManager, assetManager); + } + + private ProcessorNode lookupProcessorNode(final String versionedComponentId) { + for (final ProcessorNode processorNode : processGroup.getProcessors()) { + final Optional versionedId = processorNode.getVersionedComponentId(); + if (versionedId.isPresent() && versionedId.get().equals(versionedComponentId)) { + return processorNode; + } + } + + return null; + } + + @Override + public Set getProcessors() { + if (processorMap.isEmpty()) { + return Collections.emptySet(); + } + + final Set processors = new HashSet<>(); + for (final VersionedProcessor versionedProcessor : processorMap.values()) { + final ProcessorNode processorNode = processGroup.getProcessor(versionedProcessor.getInstanceIdentifier()); + if (processorNode != null) { + final ProcessorFacade processorFacade = new StandaloneProcessorFacade(processorNode, versionedProcessor, processScheduler, + parameterContext, componentContextProvider, connectorLogger, extensionManager, assetManager); + processors.add(processorFacade); + } + } + + return processors; + } + + @Override + public ControllerServiceFacade getControllerService(final String id) { + final ControllerServiceNode controllerServiceNode = lookupControllerServiceNode(id); + if (controllerServiceNode == null) { + return null; + } + + final VersionedControllerService controllerService = controllerServiceMap.get(id); + if (controllerService == null) { + return null; + } + + return new StandaloneControllerServiceFacade(controllerServiceNode, controllerService, parameterContext, processScheduler, + componentContextProvider, connectorLogger, extensionManager, assetManager); + } + + private ControllerServiceNode lookupControllerServiceNode(final String versionedComponentId) { + for (final ControllerServiceNode controllerServiceNode : processGroup.getControllerServices(false)) { + final Optional versionedId = controllerServiceNode.getVersionedComponentId(); + if (versionedId.isPresent() && versionedId.get().equals(versionedComponentId)) { + return controllerServiceNode; + } + } + + return null; + } + + @Override + public Set getControllerServices() { + if (controllerServiceMap.isEmpty()) { + return Collections.emptySet(); + } + + final Set controllerServices = new HashSet<>(); + for (final VersionedControllerService versionedControllerService : controllerServiceMap.values()) { + final ControllerServiceNode controllerServiceNode = processGroup.getControllerService(versionedControllerService.getInstanceIdentifier()); + if (controllerServiceNode != null) { + final ControllerServiceFacade serviceFacade = new StandaloneControllerServiceFacade(controllerServiceNode, versionedControllerService, parameterContext, + processScheduler, componentContextProvider, connectorLogger, extensionManager, assetManager); + controllerServices.add(serviceFacade); + } + } + + return controllerServices; + } + + @Override + public Set getControllerServices(final ControllerServiceReferenceScope referenceScope, final ComponentHierarchyScope hierarchyScope) { + + final boolean recursive = (hierarchyScope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + if (referenceScope == ControllerServiceReferenceScope.INCLUDE_ALL) { + final Set facades = new HashSet<>(); + collectControllerServiceFacades(this, facades, facade -> true, recursive); + return facades; + } else { + final Set serviceNodes = lifecycle.findReferencedServices(recursive); + final Set versionedComponentIds = serviceNodes.stream() + .map(ControllerServiceNode::getVersionedComponentId) + .filter(Optional::isPresent) + .map(Optional::get) + .collect(Collectors.toSet()); + + final Set facades = new HashSet<>(); + final Predicate serviceFacadeFilter = facade -> versionedComponentIds.contains(facade.getDefinition().getIdentifier()); + collectControllerServiceFacades(this, facades, serviceFacadeFilter, recursive); + return facades; + } + } + + private void collectControllerServiceFacades(final ProcessGroupFacade group, final Set facades, final Predicate filter, + final boolean recursive) { + + for (final ControllerServiceFacade serviceFacade : group.getControllerServices()) { + if (filter.test(serviceFacade)) { + facades.add(serviceFacade); + } + } + + if (recursive) { + for (final ProcessGroupFacade childGroup : group.getProcessGroups()) { + collectControllerServiceFacades(childGroup, facades, filter, true); + } + } + } + + @Override + public ConnectionFacade getConnection(final String id) { + final Connection connection = lookupConnection(id); + if (connection == null) { + return null; + } + + final VersionedConnection versionedConnection = connectionMap.get(id); + if (versionedConnection == null) { + return null; + } + + return new StandaloneConnectionFacade(connection, versionedConnection); + } + + private Connection lookupConnection(final String versionedComponentId) { + for (final Connection connection : processGroup.getConnections()) { + final Optional versionedId = connection.getVersionedComponentId(); + if (versionedId.isPresent() && versionedId.get().equals(versionedComponentId)) { + return connection; + } + } + + return null; + } + + @Override + public Set getConnections() { + if (connectionMap.isEmpty()) { + return Collections.emptySet(); + } + + final Set connections = new HashSet<>(); + for (final VersionedConnection versionedConnection : connectionMap.values()) { + final Connection connection = processGroup.getConnection(versionedConnection.getInstanceIdentifier()); + if (connection != null) { + connections.add(new StandaloneConnectionFacade(connection, versionedConnection)); + } + } + + return connections; + } + + @Override + public ProcessGroupFacade getProcessGroup(final String id) { + final ProcessGroup childProcessGroup = lookupProcessGroup(processGroup, id); + if (childProcessGroup == null) { + return null; + } + + final VersionedProcessGroup versionedProcessGroup = processGroupMap.get(id); + if (versionedProcessGroup == null) { + return null; + } + + return new StandaloneProcessGroupFacade(childProcessGroup, versionedProcessGroup, processScheduler, parameterContext, + controllerServiceProvider, componentContextProvider, connectorLogger, extensionManager, assetManager); + } + + private ProcessGroup lookupProcessGroup(final ProcessGroup start, final String versionedComponentId) { + for (final ProcessGroup childProcessGroup : start.getProcessGroups()) { + final Optional versionedId = childProcessGroup.getVersionedComponentId(); + if (versionedId.isPresent() && versionedId.get().equals(versionedComponentId)) { + return childProcessGroup; + } + } + + for (final ProcessGroup childProcessGroup : start.getProcessGroups()) { + final ProcessGroup found = lookupProcessGroup(childProcessGroup, versionedComponentId); + if (found != null) { + return found; + } + } + + return null; + } + + @Override + public Set getProcessGroups() { + if (processGroupMap.isEmpty()) { + return Collections.emptySet(); + } + + final Set processGroups = new HashSet<>(); + for (final VersionedProcessGroup versionedProcessGroup : processGroupMap.values()) { + final ProcessGroup childProcessGroup = processGroup.findProcessGroup(versionedProcessGroup.getInstanceIdentifier()); + if (childProcessGroup != null) { + final ProcessGroupFacade groupFacade = new StandaloneProcessGroupFacade(childProcessGroup, versionedProcessGroup, processScheduler, parameterContext, + controllerServiceProvider, componentContextProvider, connectorLogger, extensionManager, assetManager); + processGroups.add(groupFacade); + } + } + + return processGroups; + } + + @Override + public QueueSize getQueueSize() { + return processGroup.getQueueSize(); + } + + @Override + public boolean isFlowEmpty() { + return processGroup.isEmpty(); + } + + @Override + public StatelessGroupLifecycle getStatelessLifecycle() { + return statelessGroupLifecycle; + } + + @Override + public ProcessGroupLifecycle getLifecycle() { + return lifecycle; + } + + @Override + public String toString() { + return "StandaloneProcessGroupFacade[id=" + flowDefinition.getIdentifier() + ", name=" + flowDefinition.getName() + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final StandaloneProcessGroupFacade that = (StandaloneProcessGroupFacade) o; + return Objects.equals(flowDefinition.getIdentifier(), that.flowDefinition.getIdentifier()); + } + + @Override + public int hashCode() { + return Objects.hashCode(flowDefinition.getIdentifier()); + } + + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + DropFlowFileSummary summary = new DropFlowFileSummary(0, 0); + + // Drop from all connections in this group + for (final ConnectionFacade connection : getConnections()) { + final DropFlowFileSummary connectionSummary = connection.dropFlowFiles(predicate); + summary = summary.add(connectionSummary); + } + + // Recursively drop from all child process groups + for (final ProcessGroupFacade childGroup : getProcessGroups()) { + final DropFlowFileSummary childSummary = childGroup.dropFlowFiles(predicate); + summary = summary.add(childSummary); + } + + return summary; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessGroupLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessGroupLifecycle.java new file mode 100644 index 000000000000..3ee3d4a641e7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessGroupLifecycle.java @@ -0,0 +1,463 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.connector.components.ComponentHierarchyScope; +import org.apache.nifi.components.connector.components.ControllerServiceReferenceScope; +import org.apache.nifi.components.connector.components.ProcessGroupLifecycle; +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.connectable.Port; +import org.apache.nifi.controller.ComponentNode; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.controller.service.ControllerServiceProvider; +import org.apache.nifi.flow.ExecutionEngine; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.groups.RemoteProcessGroup; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.function.Function; +import java.util.stream.Collectors; + +public class StandaloneProcessGroupLifecycle implements ProcessGroupLifecycle { + private static final Logger logger = LoggerFactory.getLogger(StandaloneProcessGroupLifecycle.class); + + private final ProcessGroup processGroup; + private final ControllerServiceProvider controllerServiceProvider; + private final StatelessGroupLifecycle statelessGroupLifecycle; + private final Function childGroupLifecycleFactory; + + public StandaloneProcessGroupLifecycle(final ProcessGroup processGroup, final ControllerServiceProvider controllerServiceProvider, + final StatelessGroupLifecycle statelessGroupLifecycle, final Function childGroupLifecycleFactory) { + + this.processGroup = processGroup; + this.controllerServiceProvider = controllerServiceProvider; + this.statelessGroupLifecycle = statelessGroupLifecycle; + this.childGroupLifecycleFactory = childGroupLifecycleFactory; + } + + @Override + public CompletableFuture enableControllerServices(final ControllerServiceReferenceScope referenceScope, final ComponentHierarchyScope hierarchyScope) { + final boolean recursive = (hierarchyScope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + final Set controllerServices = (referenceScope == ControllerServiceReferenceScope.INCLUDE_ALL) + ? processGroup.findAllControllerServices() : findReferencedServices(recursive); + return enableControllerServices(controllerServices); + } + + public Set findReferencedServices(final boolean recursive) { + final Set referencedServices = new HashSet<>(); + collectReferencedServices(processGroup, referencedServices, recursive); + return referencedServices; + } + + private void collectReferencedServices(final ProcessGroup group, final Set referencedServices, final boolean recursive) { + for (final ProcessorNode processor : group.getProcessors()) { + final ValidationContext validationContext = createValidationContext(processor); + + for (final PropertyDescriptor descriptor : processor.getPropertyDescriptors()) { + if (descriptor.getControllerServiceDefinition() == null) { + continue; + } + + final String serviceId = processor.getEffectivePropertyValue(descriptor); + if (serviceId == null) { + continue; + } + + // Skip properties whose dependencies are not satisfied, as the property is not relevant to the component's functionality + if (!validationContext.isDependencySatisfied(descriptor, processor::getPropertyDescriptor)) { + logger.debug("Not enabling Controller Service {} because it is referenced by {} property of {} whose dependencies are not satisfied", + serviceId, descriptor.getName(), processor); + continue; + } + + final ControllerServiceNode serviceNode = controllerServiceProvider.getControllerServiceNode(serviceId); + if (serviceNode == null) { + continue; + } + + logger.debug("Marking {} as a Referenced Controller Service because it is referenced by {} property of {}", + serviceNode, descriptor.getName(), processor); + referencedServices.add(serviceNode); + } + } + + // Transitively collect services required by the already-referenced services. + // ControllerServiceNode.getRequiredControllerServices() already filters based on satisfied property dependencies. + while (true) { + final Set newlyAddedServices = new HashSet<>(); + for (final ControllerServiceNode service : referencedServices) { + for (final ControllerServiceNode requiredService : service.getRequiredControllerServices()) { + if (!referencedServices.contains(requiredService)) { + logger.debug("Marking {} as a Referenced Controller Service because it is required by {}", + requiredService, service); + newlyAddedServices.add(requiredService); + } + } + } + + referencedServices.addAll(newlyAddedServices); + if (newlyAddedServices.isEmpty()) { + break; + } + } + + if (recursive) { + for (final ProcessGroup childGroup : group.getProcessGroups()) { + collectReferencedServices(childGroup, referencedServices, true); + } + } + } + + private ValidationContext createValidationContext(final ComponentNode component) { + final Map effectivePropertyValues = new LinkedHashMap<>(); + for (final Map.Entry entry : component.getEffectivePropertyValues().entrySet()) { + effectivePropertyValues.put(entry.getKey().getName(), entry.getValue()); + } + + return component.createValidationContext(effectivePropertyValues, component.getAnnotationData(), component.getParameterLookup(), false); + } + + @Override + public CompletableFuture enableControllerServices(final Collection collection) { + final Set serviceNodes = findControllerServices(collection); + return enableControllerServices(serviceNodes); + } + + private CompletableFuture enableControllerServices(final Set serviceNodes) { + if (serviceNodes.isEmpty()) { + return CompletableFuture.completedFuture(null); + } + + // If any service is not currently valid, perform validation again to ensure that the status is up to date. + for (final ControllerServiceNode serviceNode : serviceNodes) { + final ValidationStatus validationStatus = serviceNode.getValidationStatus(); + if (validationStatus != ValidationStatus.VALID) { + serviceNode.performValidation(); + } + } + + return controllerServiceProvider.enableControllerServicesAsync(serviceNodes); + } + + private Set findControllerServices(final Collection serviceIds) { + return processGroup.findAllControllerServices().stream() + .filter(service -> service.getVersionedComponentId().isPresent()) + .filter(service -> serviceIds.contains(service.getVersionedComponentId().get())) + .collect(Collectors.toSet()); + } + + @Override + public CompletableFuture disableControllerServices(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + final Set controllerServices = recursive ? processGroup.findAllControllerServices() : processGroup.getControllerServices(false); + return disableControllerServices(controllerServices); + } + + private CompletableFuture disableControllerServices(final Set serviceNodes) { + if (serviceNodes.isEmpty()) { + return CompletableFuture.completedFuture(null); + } + + return controllerServiceProvider.disableControllerServicesAsync(serviceNodes); + } + + @Override + public CompletableFuture disableControllerServices(final Collection collection) { + final Set serviceNodes = findControllerServices(collection); + return disableControllerServices(serviceNodes); + } + + @Override + public CompletableFuture startProcessors(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + final Collection processors = recursive ? processGroup.findAllProcessors() : processGroup.getProcessors(); + final List> startFutures = new ArrayList<>(); + for (final ProcessorNode processor : processors) { + // If Processor is not valid, perform validation again to ensure that the status is up to date. + final ValidationStatus validationStatus = processor.getValidationStatus(); + if (validationStatus != ValidationStatus.VALID) { + processor.performValidation(); + } + + if (processor.getScheduledState() == ScheduledState.DISABLED) { + logger.debug("Not starting Processor {} because it is disabled", processor); + continue; + } + + startFutures.add(processor.getProcessGroup().startProcessor(processor, true)); + } + + return CompletableFuture.allOf(startFutures.toArray(new CompletableFuture[0])); + } + + @Override + public CompletableFuture start(final ControllerServiceReferenceScope serviceReferenceScope) { + if (processGroup.resolveExecutionEngine() == ExecutionEngine.STATELESS) { + return statelessGroupLifecycle.start(); + } + + final CompletableFuture result = new CompletableFuture<>(); + + Thread.startVirtualThread(() -> { + try { + enableControllerServices(serviceReferenceScope, ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + startPorts(ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + startRemoteProcessGroups(ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + startProcessors(ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + + final List> childGroupFutures = new ArrayList<>(); + for (final ProcessGroup childGroup : processGroup.getProcessGroups()) { + final String childGroupId = childGroup.getVersionedComponentId().orElse(null); + if (childGroupId == null) { + logger.warn("Encountered child Process Group {} without a Versioned Component ID. Skipping start of child group.", childGroup.getIdentifier()); + continue; + } + + final ProcessGroupLifecycle childLifecycle = childGroupLifecycleFactory.apply(childGroupId); + childGroupFutures.add(childLifecycle.start(serviceReferenceScope)); + } + + CompletableFuture.allOf(childGroupFutures.toArray(new CompletableFuture[0])).get(); + result.complete(null); + } catch (final Exception e) { + result.completeExceptionally(e); + } + }); + + return result; + } + + @Override + public CompletableFuture startPorts(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + logger.debug("{} starting all ports", this); + final Collection inputPorts = recursive ? processGroup.findAllInputPorts() : processGroup.getInputPorts(); + for (final Port inputPort : inputPorts) { + inputPort.getProcessGroup().startInputPort(inputPort); + } + + final Collection outputPorts = recursive ? processGroup.findAllOutputPorts() : processGroup.getOutputPorts(); + for (final Port outputPort : outputPorts) { + outputPort.getProcessGroup().startOutputPort(outputPort); + } + + logger.info("{} started all ports", this); + return CompletableFuture.completedFuture(null); + } + + @Override + public CompletableFuture stopPorts(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + logger.debug("{} stopping all ports", this); + + final Collection inputPorts = recursive ? processGroup.findAllInputPorts() : processGroup.getInputPorts(); + for (final Port inputPort : inputPorts) { + inputPort.getProcessGroup().stopInputPort(inputPort); + } + + final Collection outputPorts = recursive ? processGroup.findAllOutputPorts() : processGroup.getOutputPorts(); + for (final Port outputPort : outputPorts) { + outputPort.getProcessGroup().stopOutputPort(outputPort); + } + + logger.info("{} stopped all ports", this); + return CompletableFuture.completedFuture(null); + } + + @Override + public CompletableFuture startRemoteProcessGroups(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + logger.debug("{} starting all Remote Process Groups", this); + + final Collection rpgs = recursive ? processGroup.findAllRemoteProcessGroups() : processGroup.getRemoteProcessGroups(); + for (final RemoteProcessGroup rpg : rpgs) { + rpg.startTransmitting(); + } + + logger.info("{} started all Remote Process Groups", this); + return CompletableFuture.completedFuture(null); + } + + @Override + public CompletableFuture stopRemoteProcessGroups(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + logger.debug("{} stopping all Remote Process Groups", this); + final List> stopFutures = new ArrayList<>(); + + final Collection rpgs = recursive ? processGroup.findAllRemoteProcessGroups() : processGroup.getRemoteProcessGroups(); + for (final RemoteProcessGroup rpg : rpgs) { + stopFutures.add(rpg.stopTransmitting()); + } + + logger.info("{} stopped all Remote Process Groups", this); + return CompletableFuture.allOf(stopFutures.toArray(new CompletableFuture[0])); + } + + @Override + public CompletableFuture startStatelessGroups(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + logger.debug("{} starting all Stateless Process Groups", this); + final List> startFutures = new ArrayList<>(); + + final Collection processGroups = processGroup.getProcessGroups(); + for (final ProcessGroup childGroup : processGroups) { + final String childGroupId = childGroup.getVersionedComponentId().orElse(null); + if (childGroupId == null) { + logger.warn("Encountered stateless child Process Group {} without a Versioned Component ID. Skipping start.", childGroup.getIdentifier()); + continue; + } + + final ProcessGroupLifecycle childLifecycle = childGroupLifecycleFactory.apply(childGroupId); + + if (childGroup.resolveExecutionEngine() == ExecutionEngine.STATELESS) { + startFutures.add(childLifecycle.start(ControllerServiceReferenceScope.INCLUDE_REFERENCED_SERVICES_ONLY)); + } else if (recursive) { + startFutures.add(childLifecycle.startStatelessGroups(ComponentHierarchyScope.INCLUDE_CHILD_GROUPS)); + } + } + + logger.info("{} started all Stateless Process Groups", this); + return CompletableFuture.allOf(startFutures.toArray(new CompletableFuture[0])); + } + + @Override + public CompletableFuture stopStatelessGroups(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + logger.debug("{} stopping all Stateless Process Groups", this); + final List> stopFutures = new ArrayList<>(); + + for (final ProcessGroup childGroup : processGroup.getProcessGroups()) { + final String childGroupId = childGroup.getVersionedComponentId().orElse(null); + if (childGroupId == null) { + logger.warn("Encountered stateless child Process Group {} without a Versioned Component ID. Skipping stop.", childGroup.getIdentifier()); + continue; + } + + final ProcessGroupLifecycle childLifecycle = childGroupLifecycleFactory.apply(childGroupId); + + if (childGroup.resolveExecutionEngine() == ExecutionEngine.STATELESS) { + stopFutures.add(childLifecycle.stop()); + } else if (recursive) { + stopFutures.add(childLifecycle.stopStatelessGroups(ComponentHierarchyScope.INCLUDE_CHILD_GROUPS)); + } + } + + logger.info("{} stopped all Stateless Process Groups", this); + return CompletableFuture.allOf(stopFutures.toArray(new CompletableFuture[0])); + } + + @Override + public CompletableFuture stop() { + logger.debug("Stopping Process Group {}", processGroup.getIdentifier()); + if (processGroup.resolveExecutionEngine() == ExecutionEngine.STATELESS) { + return statelessGroupLifecycle.stop(); + } + + final CompletableFuture result = new CompletableFuture<>(); + + Thread.startVirtualThread(() -> { + try { + stopProcessors(ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + stopChildren().get(); + stopPorts(ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + stopRemoteProcessGroups(ComponentHierarchyScope.IMMEDIATE_GROUP_ONLY).get(); + disableControllerServices(ComponentHierarchyScope.INCLUDE_CHILD_GROUPS).get(); + + logger.info("Stopped Process Group {}", processGroup.getIdentifier()); + result.complete(null); + } catch (final Exception e) { + result.completeExceptionally(e); + } + }); + + return result; + } + + private CompletableFuture stopChildren() { + final List> childGroupFutures = new ArrayList<>(); + for (final ProcessGroup childGroup : processGroup.getProcessGroups()) { + final String childGroupId = childGroup.getVersionedComponentId().orElse(null); + if (childGroupId == null) { + logger.warn("Encountered child Process Group {} without a Versioned Component ID. Skipping stop of child group.", childGroup.getIdentifier()); + continue; + } + + final ProcessGroupLifecycle childLifecycle = childGroupLifecycleFactory.apply(childGroupId); + final CompletableFuture childFuture = childLifecycle.stop(); + childGroupFutures.add(childFuture); + } + + return CompletableFuture.allOf(childGroupFutures.toArray(new CompletableFuture[0])); + } + + @Override + public CompletableFuture stopProcessors(final ComponentHierarchyScope scope) { + final boolean recursive = (scope == ComponentHierarchyScope.INCLUDE_CHILD_GROUPS); + final Collection processors = recursive ? processGroup.findAllProcessors() : processGroup.getProcessors(); + final List> stopFutures = new ArrayList<>(); + for (final ProcessorNode processor : processors) { + final ScheduledState processorState = processor.getScheduledState(); + if (processorState == ScheduledState.DISABLED || processorState == ScheduledState.STOPPED) { + logger.debug("Not stopping Processor {} because its state is {}", processor, processorState); + continue; + } + + final CompletableFuture stopFuture = processor.getProcessGroup().stopProcessor(processor); + stopFutures.add(stopFuture); + } + + return CompletableFuture.allOf(stopFutures.toArray(new CompletableFuture[0])); + } + + @Override + public int getActiveThreadCount() { + return getActiveThreadCount(processGroup); + } + + private int getActiveThreadCount(final ProcessGroup group) { + int total = 0; + for (final ProcessorNode processor : group.getProcessors()) { + total += processor.getActiveThreadCount(); + } + for (final ProcessGroup childGroup : group.getProcessGroups()) { + total += getActiveThreadCount(childGroup); + } + return total; + } + + @Override + public String toString() { + return "StandaloneProcessGroupLifecycle[" + + "processGroup=" + processGroup.getIdentifier() + + "]"; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessorFacade.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessorFacade.java new file mode 100644 index 000000000000..3de26ad4431a --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessorFacade.java @@ -0,0 +1,253 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.ConnectorParameterLookup; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.components.ProcessorLifecycle; +import org.apache.nifi.components.validation.DisabledServiceValidationResult; +import org.apache.nifi.components.validation.ValidationState; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterLookup; +import org.apache.nifi.processor.ProcessContext; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +public class StandaloneProcessorFacade implements ProcessorFacade { + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) + .setDefaultPropertyInclusion(JsonInclude.Include.NON_NULL); + + private final ProcessorNode processorNode; + private final VersionedProcessor versionedProcessor; + private final ParameterContext parameterContext; + private final ProcessorLifecycle lifecycle; + private final ComponentContextProvider componentContextProvider; + private final ComponentLog connectorLogger; + private final ExtensionManager extensionManager; + private final AssetManager assetManager; + + public StandaloneProcessorFacade(final ProcessorNode processorNode, final VersionedProcessor versionedProcessor, final ProcessScheduler scheduler, + final ParameterContext parameterContext, final ComponentContextProvider componentContextProvider, final ComponentLog connectorLogger, + final ExtensionManager extensionManager, final AssetManager assetManager) { + + this.processorNode = processorNode; + this.versionedProcessor = versionedProcessor; + this.parameterContext = parameterContext; + this.componentContextProvider = componentContextProvider; + this.connectorLogger = connectorLogger; + this.extensionManager = extensionManager; + this.assetManager = assetManager; + + this.lifecycle = new StandaloneProcessorLifecycle(processorNode, scheduler); + } + + @Override + public VersionedProcessor getDefinition() { + return versionedProcessor; + } + + @Override + public ProcessorLifecycle getLifecycle() { + return lifecycle; + } + + @Override + public List validate() { + final ValidationStatus status = processorNode.performValidation(); + if (status == ValidationStatus.VALID) { + return Collections.emptyList(); + } + + return processorNode.getValidationErrors().stream() + .filter(result -> !result.isValid()) + .filter(result -> !DisabledServiceValidationResult.isMatch(result)) + .toList(); + } + + @Override + public List validate(final Map propertyValues) { + final ValidationContext validationContext = componentContextProvider.createValidationContext(processorNode, propertyValues, parameterContext); + final ValidationState validationState = processorNode.performValidation(validationContext); + final List validationErrors = validationState.getValidationErrors().stream() + .filter(result -> !result.isValid()) + .filter(result -> !DisabledServiceValidationResult.isMatch(result)) + .toList(); + + return validationErrors; + } + + @Override + public List verify(final Map propertyValues, final Map attributes) { + final ProcessContext processContext = componentContextProvider.createProcessContext(processorNode, propertyValues, parameterContext); + return processorNode.verifyConfiguration(processContext, connectorLogger, attributes, extensionManager, parameterContext); + } + + @Override + public List verify(final Map propertyValues, final VersionedParameterContext versionedParameterContext, final Map attributes) { + final ParameterLookup parameterLookup = new ConnectorParameterLookup(List.of(versionedParameterContext), assetManager); + final ProcessContext processContext = componentContextProvider.createProcessContext(processorNode, propertyValues, parameterLookup); + return processorNode.verifyConfiguration(processContext, connectorLogger, attributes, extensionManager, parameterContext); + } + + @Override + public List verify(final VersionedExternalFlow versionedExternalFlow, final Map attributes) { + final VersionedProcessGroup parentGroup = findParentGroup(versionedExternalFlow.getFlowContents()); + final VersionedProcessor providedProcessor = findVersionedProcessor(parentGroup); + if (providedProcessor == null) { + throw new IllegalArgumentException("Cannot verify Processor with ID " + this.versionedProcessor.getIdentifier() + " because it does not exist in the provided flow"); + } + final ParameterLookup parameterLookup = new ConnectorParameterLookup(versionedExternalFlow.getParameterContexts().values(), assetManager); + + final ProcessContext processContext = componentContextProvider.createProcessContext(processorNode, providedProcessor.getProperties(), parameterLookup); + return processorNode.verifyConfiguration(processContext, connectorLogger, attributes, extensionManager, parameterLookup); + } + + private VersionedProcessGroup findParentGroup(final VersionedProcessGroup group) { + for (final VersionedProcessor processor : group.getProcessors()) { + if (processor.getIdentifier().equals(versionedProcessor.getIdentifier())) { + return group; + } + } + + for (final VersionedProcessGroup childGroup : group.getProcessGroups()) { + final VersionedProcessGroup serviceGroup = findParentGroup(childGroup); + if (serviceGroup != null) { + return serviceGroup; + } + } + + return null; + } + + private VersionedProcessor findVersionedProcessor(final VersionedProcessGroup group) { + if (group == null) { + return null; + } + + final VersionedProcessGroup parentGroup = findParentGroup(group); + if (parentGroup == null) { + return null; + } + + for (final VersionedProcessor processor : group.getProcessors()) { + if (processor.getIdentifier().equals(processorNode.getVersionedComponentId().orElseThrow())) { + return processor; + } + } + + return null; + } + + + @Override + public Object invokeConnectorMethod(final String methodName, final Map arguments) throws InvocationFailedException { + final Map jsonArguments = serializeArgumentsToJson(arguments); + final ProcessContext processContext = componentContextProvider.createProcessContext(processorNode, parameterContext); + final String jsonResult = processorNode.invokeConnectorMethod(methodName, jsonArguments, processContext); + if (jsonResult == null) { + return null; + } + + try { + return OBJECT_MAPPER.readValue(jsonResult, Object.class); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to deserialize return value from Connector Method '" + methodName + "'", e); + } + } + + @Override + public T invokeConnectorMethod(final String methodName, final Map arguments, final Class returnType) throws InvocationFailedException { + final Map jsonArguments = serializeArgumentsToJson(arguments); + final ProcessContext processContext = componentContextProvider.createProcessContext(processorNode, parameterContext); + final String jsonResult = processorNode.invokeConnectorMethod(methodName, jsonArguments, processContext); + if (jsonResult == null) { + return null; + } + + try { + return OBJECT_MAPPER.readValue(jsonResult, returnType); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to deserialize return value from Connector Method '" + methodName + "'", e); + } + } + + @Override + public String toString() { + final String type = versionedProcessor.getType(); + final String simpleType = type == null ? null : type.substring(type.lastIndexOf('.') + 1); + return "StandaloneProcessorFacade[id=" + versionedProcessor.getIdentifier() + ", name=" + versionedProcessor.getName() + ", type=" + simpleType + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final StandaloneProcessorFacade that = (StandaloneProcessorFacade) o; + return Objects.equals(versionedProcessor.getIdentifier(), that.versionedProcessor.getIdentifier()); + } + + @Override + public int hashCode() { + return Objects.hashCode(versionedProcessor.getIdentifier()); + } + + private Map serializeArgumentsToJson(final Map arguments) throws InvocationFailedException { + final Map jsonArguments = new HashMap<>(); + + for (final Map.Entry entry : arguments.entrySet()) { + if (entry.getValue() == null) { + jsonArguments.put(entry.getKey(), null); + } else { + try { + jsonArguments.put(entry.getKey(), OBJECT_MAPPER.writeValueAsString(entry.getValue())); + } catch (final JsonProcessingException e) { + throw new InvocationFailedException("Failed to serialize argument '" + entry.getKey() + "' to JSON", e); + } + } + } + + return jsonArguments; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessorLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessorLifecycle.java new file mode 100644 index 000000000000..bd83cbe3e70c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneProcessorLifecycle.java @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.connector.components.ProcessorLifecycle; +import org.apache.nifi.components.connector.components.ProcessorState; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.lifecycle.ProcessorStopLifecycleMethods; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.CompletableFuture; + +public class StandaloneProcessorLifecycle implements ProcessorLifecycle { + private static final Logger logger = LoggerFactory.getLogger(StandaloneProcessorLifecycle.class); + private final ProcessorNode processorNode; + private final ProcessScheduler processScheduler; + + public StandaloneProcessorLifecycle(final ProcessorNode processorNode, final ProcessScheduler scheduler) { + this.processorNode = processorNode; + this.processScheduler = scheduler; + } + + @Override + public ProcessorState getState() { + final ScheduledState scheduledState = processorNode.getScheduledState(); + return switch (scheduledState) { + case DISABLED -> ProcessorState.DISABLED; + case RUNNING, RUN_ONCE -> ProcessorState.RUNNING; + case STOPPED -> processorNode.getActiveThreadCount() == 0 ? ProcessorState.STOPPED : ProcessorState.STOPPING; + case STOPPING -> ProcessorState.STOPPING; + case STARTING -> ProcessorState.STARTING; + }; + } + + @Override + public int getActiveThreadCount() { + return processorNode.getActiveThreadCount(); + } + + @Override + public void terminate() { + try { + processScheduler.terminateProcessor(processorNode); + } catch (final Exception e) { + logger.warn("Failed to terminate processor {}", processorNode, e); + } + } + + @Override + public CompletableFuture stop() { + return processScheduler.stopProcessor(processorNode, ProcessorStopLifecycleMethods.TRIGGER_ALL); + } + + @Override + public CompletableFuture start() { + // Perform validation if necessary before starting. + final ValidationStatus validationStatus = processorNode.getValidationStatus(); + if (validationStatus != ValidationStatus.VALID) { + processorNode.performValidation(); + } + + return processScheduler.startProcessor(processorNode, false); + } + + @Override + public void disable() { + processScheduler.disableProcessor(processorNode); + } + + @Override + public void enable() { + processScheduler.enableProcessor(processorNode); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneStatelessGroupLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneStatelessGroupLifecycle.java new file mode 100644 index 000000000000..f29287b32390 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/facades/standalone/StandaloneStatelessGroupLifecycle.java @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.groups.StatelessGroupNode; + +import java.util.concurrent.CompletableFuture; + +public class StandaloneStatelessGroupLifecycle implements StatelessGroupLifecycle { + private final StatelessGroupNode statelessGroupNode; + private final ProcessScheduler processScheduler; + + public StandaloneStatelessGroupLifecycle(final ProcessGroup processGroup, final ProcessScheduler processScheduler) { + this.statelessGroupNode = processGroup.getStatelessGroupNode().orElseThrow(() -> new IllegalStateException("Process Group is not configured to run using the Stateless Execution Engine")); + this.processScheduler = processScheduler; + } + + @Override + public CompletableFuture start() { + statelessGroupNode.setDesiredState(ScheduledState.RUNNING); + return processScheduler.startStatelessGroup(statelessGroupNode); + } + + @Override + public CompletableFuture stop() { + statelessGroupNode.setDesiredState(ScheduledState.STOPPED); + return processScheduler.stopStatelessGroup(statelessGroupNode); + } + + // TODO: Stateless Group does not currently support termination. + @Override + public CompletableFuture terminate() { + return CompletableFuture.completedFuture(null); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/validation/TriggerValidationTask.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/validation/TriggerValidationTask.java index 69c32bf74687..d05033a8b4d3 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/validation/TriggerValidationTask.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/validation/TriggerValidationTask.java @@ -17,6 +17,7 @@ package org.apache.nifi.components.validation; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.controller.ComponentNode; import org.apache.nifi.controller.flow.FlowManager; import org.slf4j.Logger; @@ -61,6 +62,10 @@ public void run() { for (final ComponentNode node : flowManager.getAllFlowRegistryClients()) { validationTrigger.trigger(node); } + + for (final ConnectorNode connector : flowManager.getAllConnectors()) { + connector.validateComponents(validationTrigger); + } } catch (final Throwable t) { logger.error("Encountered unexpected error when attempting to validate components", t); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/ExtensionBuilder.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/ExtensionBuilder.java index 82cf84190cd4..45ef9980d4b2 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/ExtensionBuilder.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/ExtensionBuilder.java @@ -21,10 +21,32 @@ import org.apache.nifi.annotation.behavior.RequiresInstanceClassLoading; import org.apache.nifi.annotation.behavior.SupportsBatching; import org.apache.nifi.annotation.configuration.DefaultSettings; +import org.apache.nifi.authorization.Resource; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.resource.ResourceFactory; import org.apache.nifi.bundle.Bundle; import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.components.ConfigurableComponent; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.ComponentBundleLookup; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorDetails; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.ConnectorStateTransition; +import org.apache.nifi.components.connector.ConnectorValidationTrigger; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FlowContextFactory; +import org.apache.nifi.components.connector.FrameworkConnectorInitializationContext; +import org.apache.nifi.components.connector.FrameworkConnectorInitializationContextBuilder; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.GhostConnector; +import org.apache.nifi.components.connector.MutableConnectorConfigurationContext; +import org.apache.nifi.components.connector.StandardConnectorNode; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.components.connector.StringLiteralValue; import org.apache.nifi.components.state.StateManager; import org.apache.nifi.components.state.StateManagerProvider; import org.apache.nifi.components.validation.ValidationTrigger; @@ -52,6 +74,7 @@ import org.apache.nifi.flowanalysis.FlowAnalysisRule; import org.apache.nifi.flowanalysis.FlowAnalysisRuleInitializationContext; import org.apache.nifi.flowanalysis.GhostFlowAnalysisRule; +import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.logging.ComponentLog; import org.apache.nifi.logging.LoggingContext; import org.apache.nifi.logging.StandardLoggingContext; @@ -89,8 +112,10 @@ import java.lang.reflect.Proxy; import java.net.URL; import java.util.Collections; +import java.util.HashMap; import java.util.HashSet; import java.util.List; +import java.util.Map; import java.util.Set; import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; @@ -119,6 +144,13 @@ public class ExtensionBuilder { private SSLContext systemSslContext; private PythonBridge pythonBridge; private VerifiableComponentFactory verifiableComponentFactory; + private ProcessGroup managedProcessGroup; + private FlowContextFactory flowContextFactory; + private MutableConnectorConfigurationContext activeConfigurationContext; + private ConnectorStateTransition connectorStateTransition; + private FrameworkConnectorInitializationContextBuilder connectorInitializationContextBuilder; + private ConnectorValidationTrigger connectorValidationTrigger; + private ComponentBundleLookup componentBundleLookup; public ExtensionBuilder type(final String type) { this.type = type; @@ -193,12 +225,12 @@ public ExtensionBuilder stateManagerProvider(final StateManagerProvider stateMan return this; } - public ExtensionBuilder ruleViolationsManager(RuleViolationsManager ruleViolationsManager) { + public ExtensionBuilder ruleViolationsManager(final RuleViolationsManager ruleViolationsManager) { this.ruleViolationsManager = ruleViolationsManager; return this; } - public ExtensionBuilder flowAnalyzer(FlowAnalyzer flowAnalyzer) { + public ExtensionBuilder flowAnalyzer(final FlowAnalyzer flowAnalyzer) { this.flowAnalyzer = flowAnalyzer; return this; } @@ -228,31 +260,49 @@ public ExtensionBuilder verifiableComponentFactory(final VerifiableComponentFact return this; } + public ExtensionBuilder managedProcessGroup(final ProcessGroup managedProcessGroup) { + this.managedProcessGroup = managedProcessGroup; + return this; + } + + public ExtensionBuilder activeConfigurationContext(final MutableConnectorConfigurationContext configurationContext) { + this.activeConfigurationContext = configurationContext; + return this; + } + + public ExtensionBuilder flowContextFactory(final FlowContextFactory flowContextFactory) { + this.flowContextFactory = flowContextFactory; + return this; + } + + public ExtensionBuilder connectorStateTransition(final ConnectorStateTransition connectorStateTransition) { + this.connectorStateTransition = connectorStateTransition; + return this; + } + + public ExtensionBuilder connectorInitializationContextBuilder(final FrameworkConnectorInitializationContextBuilder contextBuilder) { + this.connectorInitializationContextBuilder = contextBuilder; + return this; + } + + public ExtensionBuilder connectorValidationTrigger(final ConnectorValidationTrigger connectorValidationTrigger) { + this.connectorValidationTrigger = connectorValidationTrigger; + return this; + } + + public ExtensionBuilder componentBundleLookup(final ComponentBundleLookup componentBundleLookup) { + this.componentBundleLookup = componentBundleLookup; + return this; + } + public ProcessorNode buildProcessor() { - if (identifier == null) { - throw new IllegalStateException("Processor ID must be specified"); - } - if (type == null) { - throw new IllegalStateException("Processor Type must be specified"); - } - if (bundleCoordinate == null) { - throw new IllegalStateException("Bundle Coordinate must be specified"); - } - if (extensionManager == null) { - throw new IllegalStateException("Extension Manager must be specified"); - } - if (serviceProvider == null) { - throw new IllegalStateException("Controller Service Provider must be specified"); - } - if (nodeTypeProvider == null) { - throw new IllegalStateException("Node Type Provider must be specified"); - } - if (reloadComponent == null) { - throw new IllegalStateException("Reload Component must be specified"); - } - if (verifiableComponentFactory == null) { - throw new IllegalStateException("Verifiable Component Factory must be specified"); - } + requireNonNull(identifier, "Processor ID"); + requireNonNull(type, "Processor Type"); + requireNonNull(bundleCoordinate, "Bundle Coordinate"); + requireNonNull(extensionManager, "Extension Manager"); + requireNonNull(serviceProvider, "Controller Service Provider"); + requireNonNull(nodeTypeProvider, "Node Type Provider"); + requireNonNull(reloadComponent, "Reload Component"); boolean creationSuccessful = true; final StandardLoggingContext loggingContext = new StandardLoggingContext(); @@ -260,10 +310,7 @@ public ProcessorNode buildProcessor() { try { loggableComponent = createLoggableProcessor(loggingContext); } catch (final ProcessorInstantiationException pie) { - logger.error("Could not create Processor of type {} from {} for ID {} due to: {}; creating \"Ghost\" implementation", type, bundleCoordinate, identifier, pie.getMessage()); - if (logger.isDebugEnabled()) { - logger.debug(pie.getMessage(), pie); - } + logger.error("Could not create Processor of type {} from {} for ID {} due to: {}; creating \"Ghost\" implementation", type, bundleCoordinate, identifier, pie.getMessage(), pie); final GhostProcessor ghostProc = new GhostProcessor(); ghostProc.setIdentifier(identifier); ghostProc.setCanonicalClassName(type); @@ -287,30 +334,14 @@ public ProcessorNode buildProcessor() { } public FlowRegistryClientNode buildFlowRegistryClient() { - if (identifier == null) { - throw new IllegalStateException("ReportingTask ID must be specified"); - } - if (type == null) { - throw new IllegalStateException("ReportingTask Type must be specified"); - } - if (bundleCoordinate == null) { - throw new IllegalStateException("Bundle Coordinate must be specified"); - } - if (serviceProvider == null) { - throw new IllegalStateException("Controller Service Provider must be specified"); - } - if (extensionManager == null) { - throw new IllegalStateException("Extension Manager must be specified"); - } - if (nodeTypeProvider == null) { - throw new IllegalStateException("Node Type Provider must be specified"); - } - if (reloadComponent == null) { - throw new IllegalStateException("Reload Component must be specified"); - } - if (flowController == null) { - throw new IllegalStateException("FlowController must be specified"); - } + requireNonNull(identifier, "Flow Registry Client ID"); + requireNonNull(type, "Flow Registry Client Type"); + requireNonNull(bundleCoordinate, "Bundle Coordinate"); + requireNonNull(serviceProvider, "Controller Service Provider"); + requireNonNull(extensionManager, "Extension Manager"); + requireNonNull(nodeTypeProvider, "Node Type Provider"); + requireNonNull(reloadComponent, "Reload Component"); + requireNonNull(flowController, "FlowController"); boolean creationSuccessful = true; LoggableComponent loggableComponent; @@ -331,30 +362,14 @@ public FlowRegistryClientNode buildFlowRegistryClient() { } public ReportingTaskNode buildReportingTask() { - if (identifier == null) { - throw new IllegalStateException("ReportingTask ID must be specified"); - } - if (type == null) { - throw new IllegalStateException("ReportingTask Type must be specified"); - } - if (bundleCoordinate == null) { - throw new IllegalStateException("Bundle Coordinate must be specified"); - } - if (extensionManager == null) { - throw new IllegalStateException("Extension Manager must be specified"); - } - if (serviceProvider == null) { - throw new IllegalStateException("Controller Service Provider must be specified"); - } - if (nodeTypeProvider == null) { - throw new IllegalStateException("Node Type Provider must be specified"); - } - if (reloadComponent == null) { - throw new IllegalStateException("Reload Component must be specified"); - } - if (flowController == null) { - throw new IllegalStateException("FlowController must be specified"); - } + requireNonNull(identifier, "ReportingTask ID"); + requireNonNull(type, "ReportingTask Type"); + requireNonNull(bundleCoordinate, "Bundle Coordinate"); + requireNonNull(extensionManager, "Extension Manager"); + requireNonNull(serviceProvider, "Controller Service Provider"); + requireNonNull(nodeTypeProvider, "Node Type Provider"); + requireNonNull(reloadComponent, "Reload Component"); + requireNonNull(flowController, "FlowController"); boolean creationSuccessful = true; LoggableComponent loggableComponent; @@ -377,30 +392,14 @@ public ReportingTaskNode buildReportingTask() { } public ParameterProviderNode buildParameterProvider() { - if (identifier == null) { - throw new IllegalStateException("ParameterProvider ID must be specified"); - } - if (type == null) { - throw new IllegalStateException("ParameterProvider Type must be specified"); - } - if (bundleCoordinate == null) { - throw new IllegalStateException("Bundle Coordinate must be specified"); - } - if (extensionManager == null) { - throw new IllegalStateException("Extension Manager must be specified"); - } - if (serviceProvider == null) { - throw new IllegalStateException("Controller Service Provider must be specified"); - } - if (nodeTypeProvider == null) { - throw new IllegalStateException("Node Type Provider must be specified"); - } - if (reloadComponent == null) { - throw new IllegalStateException("Reload Component must be specified"); - } - if (flowController == null) { - throw new IllegalStateException("FlowController must be specified"); - } + requireNonNull(identifier, "ParameterProvider ID"); + requireNonNull(type, "ParameterProvider Type"); + requireNonNull(bundleCoordinate, "Bundle Coordinate"); + requireNonNull(extensionManager, "Extension Manager"); + requireNonNull(serviceProvider, "Controller Service Provider"); + requireNonNull(nodeTypeProvider, "Node Type Provider"); + requireNonNull(reloadComponent, "Reload Component"); + requireNonNull(flowController, "FlowController"); boolean creationSuccessful = true; LoggableComponent loggableComponent; @@ -464,30 +463,14 @@ public ControllerServiceNode buildControllerService() { } public FlowAnalysisRuleNode buildFlowAnalysisRuleNode() { - if (identifier == null) { - throw new IllegalStateException("FlowAnalysisRule ID must be specified"); - } - if (type == null) { - throw new IllegalStateException("FlowAnalysisRule Type must be specified"); - } - if (bundleCoordinate == null) { - throw new IllegalStateException("Bundle Coordinate must be specified"); - } - if (extensionManager == null) { - throw new IllegalStateException("Extension Manager must be specified"); - } - if (serviceProvider == null) { - throw new IllegalStateException("Controller Service Provider must be specified"); - } - if (nodeTypeProvider == null) { - throw new IllegalStateException("Node Type Provider must be specified"); - } - if (reloadComponent == null) { - throw new IllegalStateException("Reload Component must be specified"); - } - if (flowController == null) { - throw new IllegalStateException("FlowController must be specified"); - } + requireNonNull(identifier, "FlowAnalysisRule ID"); + requireNonNull(type, "FlowAnalysisRule Type"); + requireNonNull(bundleCoordinate, "Bundle Coordinate"); + requireNonNull(extensionManager, "Extension Manager"); + requireNonNull(serviceProvider, "Controller Service Provider"); + requireNonNull(nodeTypeProvider, "Node Type Provider"); + requireNonNull(reloadComponent, "Reload Component"); + requireNonNull(flowController, "FlowController"); boolean creationSuccessful = true; LoggableComponent loggableComponent; @@ -510,6 +493,145 @@ public FlowAnalysisRuleNode buildFlowAnalysisRuleNode() { return flowAnalysisRuleNode; } + public ConnectorNode buildConnector(final boolean loadInitialFlow) { + requireNonNull(identifier, "Connector ID"); + requireNonNull(type, "Connector Type"); + requireNonNull(bundleCoordinate, "Bundle Coordinate"); + requireNonNull(extensionManager, "Extension Manager"); + requireNonNull(managedProcessGroup, "Managed Process Group"); + + final Authorizable connectorsAuthorizable = new Authorizable() { + @Override + public Authorizable getParentAuthorizable() { + return null; + } + + @Override + public Resource getResource() { + return ResourceFactory.getConnectorsResource(); + } + }; + + final Connector connector; + try { + connector = createConnector(); + } catch (final Exception e) { + logger.error("Could not create Connector of type {} from {} for ID {} due to: {}; creating \"Ghost\" implementation", type, bundleCoordinate, identifier, e.getMessage(), e); + return createGhostConnectorNode(connectorsAuthorizable, e); + } + + final String componentType = connector.getClass().getSimpleName(); + final ComponentLog componentLog = new SimpleProcessLogger(identifier, connector, new StandardLoggingContext()); + final ConnectorDetails connectorDetails = new ConnectorDetails(connector, bundleCoordinate, componentLog); + + final ConnectorNode connectorNode = new StandardConnectorNode( + identifier, + flowController.getFlowManager(), + extensionManager, + connectorsAuthorizable, + connectorDetails, + componentType, + type, + activeConfigurationContext, + connectorStateTransition, + flowContextFactory, + connectorValidationTrigger, + false + ); + + try { + initializeDefaultValues(connector, connectorNode.getActiveFlowContext()); + + final FrameworkConnectorInitializationContext initContext = createConnectorInitializationContext(managedProcessGroup, componentLog); + connectorNode.initializeConnector(initContext); + } catch (final Exception e) { + logger.error("Could not initialize Connector of type {} from {} for ID {}; creating \"Ghost\" implementation", type, bundleCoordinate, identifier, e); + return createGhostConnectorNode(connectorsAuthorizable, e); + } + + if (loadInitialFlow) { + try { + connectorNode.loadInitialFlow(); + } catch (final Exception e) { + logger.error("Failed to load initial flow for {}; creating \"Ghost\" implementation", connectorNode, e); + return createGhostConnectorNode(connectorsAuthorizable, e); + } + } + + return connectorNode; + } + + private ConnectorNode createGhostConnectorNode(final Authorizable connectorsAuthorizable, final Exception cause) { + final GhostConnector ghostConnector = new GhostConnector(identifier, type, cause); + final String simpleClassName = type.contains(".") ? StringUtils.substringAfterLast(type, ".") : type; + final String componentType = "(Missing) " + simpleClassName; + final ComponentLog componentLog = new SimpleProcessLogger(identifier, ghostConnector, new StandardLoggingContext()); + final ConnectorDetails connectorDetails = new ConnectorDetails(ghostConnector, bundleCoordinate, componentLog); + + // If an instance class loader has been created for this connector, remove it because it's no longer necessary. + extensionManager.removeInstanceClassLoader(identifier); + + final ConnectorNode connectorNode = new StandardConnectorNode( + identifier, + flowController.getFlowManager(), + extensionManager, + connectorsAuthorizable, + connectorDetails, + componentType, + type, + activeConfigurationContext, + connectorStateTransition, + flowContextFactory, + connectorValidationTrigger, + true + ); + + // Initialize the ghost connector so that it can be properly configured during flow synchronization + final FrameworkConnectorInitializationContext initContext = createConnectorInitializationContext(managedProcessGroup, componentLog); + connectorNode.initializeConnector(initContext); + + return connectorNode; + } + + private void initializeDefaultValues(final Connector connector, final FrameworkFlowContext flowContext) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connector.getClass(), identifier)) { + final List configSteps = connector.getConfigurationSteps(); + + for (final ConfigurationStep step : configSteps) { + final Map defaultValues = new HashMap<>(); + + for (final ConnectorPropertyGroup propertyGroup : step.getPropertyGroups()) { + for (final ConnectorPropertyDescriptor descriptor : propertyGroup.getProperties()) { + final String name = descriptor.getName(); + final String defaultValue = descriptor.getDefaultValue(); + defaultValues.put(name, defaultValue == null ? null : new StringLiteralValue(defaultValue)); + } + } + + flowContext.getConfigurationContext().setProperties(step.getName(), new StepConfiguration(defaultValues)); + } + } + } + + private FrameworkConnectorInitializationContext createConnectorInitializationContext(final ProcessGroup managedProcessGroup, final ComponentLog componentLog) { + final String name = type.contains(".") ? StringUtils.substringAfterLast(type, ".") : type; + + return connectorInitializationContextBuilder + .identifier(identifier) + .name(name) + .componentLog(componentLog) + .secretsManager(flowController.getConnectorRepository().getSecretsManager()) + .assetManager(flowController.getConnectorAssetManager()) + .componentBundleLookup(componentBundleLookup) + .build(); + } + + private void requireNonNull(final Object value, final String fieldName) { + if (value == null) { + throw new IllegalStateException(fieldName + " must be specified"); + } + } + private ProcessorNode createProcessorNode(final LoggableComponent processor, final String componentType, final boolean extensionMissing) { final ValidationContextFactory validationContextFactory = createValidationContextFactory(serviceProvider); @@ -542,7 +664,7 @@ private ReportingTaskNode createReportingTaskNode(final LoggableComponent createLoggablePythonProcessor(final Logging } } - private LoggableComponent createLoggableComponent(Class nodeType, LoggingContext loggingContext) + private Connector createConnector() throws Exception { + final ClassLoader ctxClassLoader = Thread.currentThread().getContextClassLoader(); + try { + final Bundle bundle = extensionManager.getBundle(bundleCoordinate); + if (bundle == null) { + throw new IllegalStateException("Unable to find bundle for coordinate " + bundleCoordinate.getCoordinate()); + } + + final ClassLoader detectedClassLoader = extensionManager.createInstanceClassLoader(type, identifier, bundle, + classpathUrls == null ? Collections.emptySet() : classpathUrls, true, classloaderIsolationKey); + final Class rawClass = Class.forName(type, true, detectedClassLoader); + Thread.currentThread().setContextClassLoader(detectedClassLoader); + + final Object extensionInstance = rawClass.getDeclaredConstructor().newInstance(); + final Connector connector = (Connector) extensionInstance; + + // TODO: Initialize the Connector - this will require creating a ConnectorInitializationContext + // For now, we'll skip initialization as it requires additional infrastructure + // that may not be available at this point. The initialization should happen + // when the connector is actually started/configured. + + return connector; + } catch (final Throwable t) { + throw new Exception("Failed to create Connector of type " + type, t); + } finally { + if (ctxClassLoader != null) { + Thread.currentThread().setContextClassLoader(ctxClassLoader); + } + } + } + + private LoggableComponent createLoggableComponent(final Class nodeType, final LoggingContext loggingContext) throws ClassNotFoundException, IllegalAccessException, InstantiationException, NoSuchMethodException, InvocationTargetException { final ClassLoader ctxClassLoader = Thread.currentThread().getContextClassLoader(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java index ea40ea135bbd..14cd4f505d0c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java @@ -215,6 +215,15 @@ public void purge() { } } + @Override + public void deleteSwapFile(final String swapLocation) throws IOException { + final File swapFile = new File(swapLocation); + if (swapFile.exists() && !swapFile.delete()) { + throw new IOException("Failed to delete swap file " + swapLocation); + } + logger.debug("Deleted swap file {}", swapLocation); + } + @Override public String getQueueIdentifier(final String swapLocation) { final String filename = swapLocation.contains("/") ? StringUtils.substringAfterLast(swapLocation, "/") : swapLocation; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java index cdbbc6e2c9be..7051bab362a8 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java @@ -27,6 +27,7 @@ import org.apache.nifi.asset.StandardAssetManager; import org.apache.nifi.asset.StandardAssetManagerInitializationContext; import org.apache.nifi.asset.StandardAssetReferenceLookup; +import org.apache.nifi.asset.StandardConnectorAssetManager; import org.apache.nifi.authorization.Authorizer; import org.apache.nifi.authorization.Resource; import org.apache.nifi.authorization.resource.Authorizable; @@ -48,6 +49,22 @@ import org.apache.nifi.cluster.protocol.UnknownServiceAddressException; import org.apache.nifi.cluster.protocol.message.HeartbeatMessage; import org.apache.nifi.components.ClassLoaderAwarePythonBridge; +import org.apache.nifi.components.connector.ConnectorConfigurationProvider; +import org.apache.nifi.components.connector.ConnectorConfigurationProviderInitializationContext; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorRepositoryInitializationContext; +import org.apache.nifi.components.connector.ConnectorRequestReplicator; +import org.apache.nifi.components.connector.ConnectorValidationTrigger; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.StandardConnectorConfigurationProviderInitializationContext; +import org.apache.nifi.components.connector.StandardConnectorRepoInitializationContext; +import org.apache.nifi.components.connector.StandardConnectorRepository; +import org.apache.nifi.components.connector.StandardConnectorValidationTrigger; +import org.apache.nifi.components.connector.secrets.ParameterProviderSecretsManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.components.connector.secrets.SecretsManagerInitializationContext; +import org.apache.nifi.components.connector.secrets.StandardSecretsManagerInitializationContext; import org.apache.nifi.components.monitor.LongRunningTaskMonitor; import org.apache.nifi.components.state.StateManagerProvider; import org.apache.nifi.components.state.StateProvider; @@ -190,7 +207,7 @@ import org.apache.nifi.python.PythonBridgeInitializationContext; import org.apache.nifi.python.PythonProcessConfig; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; import org.apache.nifi.remote.HttpRemoteSiteListener; import org.apache.nifi.remote.RemoteGroupPort; @@ -266,6 +283,9 @@ public class FlowController implements ReportingTaskProvider, FlowAnalysisRulePr public static final String DEFAULT_PROVENANCE_REPO_IMPLEMENTATION = "org.apache.nifi.provenance.VolatileProvenanceRepository"; public static final String DEFAULT_SWAP_MANAGER_IMPLEMENTATION = "org.apache.nifi.controller.FileSystemSwapManager"; public static final String DEFAULT_ASSET_MANAGER_IMPLEMENTATION = StandardAssetManager.class.getName(); + public static final String DEFAULT_CONNECTOR_ASSET_MANAGER_IMPLEMENTATION = StandardConnectorAssetManager.class.getName(); + public static final String DEFAULT_CONNECTOR_REPOSITORY_IMPLEMENTATION = StandardConnectorRepository.class.getName(); + public static final String DEFAULT_SECRETS_MANAGER_IMPLEMENTATION = ParameterProviderSecretsManager.class.getName(); public static final String GRACEFUL_SHUTDOWN_PERIOD = "nifi.flowcontroller.graceful.shutdown.seconds"; public static final long DEFAULT_GRACEFUL_SHUTDOWN_SECONDS = 10; @@ -281,6 +301,7 @@ public class FlowController implements ReportingTaskProvider, FlowAnalysisRulePr private final ProvenanceRepository provenanceRepository; private final BulletinRepository bulletinRepository; private final AssetManager assetManager; + private final AssetManager connectorAssetManager; private final LifecycleStateManager lifecycleStateManager; private final StandardProcessScheduler processScheduler; private final SnippetManager snippetManager; @@ -300,6 +321,7 @@ public class FlowController implements ReportingTaskProvider, FlowAnalysisRulePr private final StateManagerProvider stateManagerProvider; private final long systemStartTime = System.currentTimeMillis(); // time at which the node was started private final RevisionManager revisionManager; + private final ConnectorRepository connectorRepository; private final ConnectionLoadBalanceServer loadBalanceServer; private final NioAsyncLoadBalanceClientRegistry loadBalanceClientRegistry; @@ -328,6 +350,7 @@ public class FlowController implements ReportingTaskProvider, FlowAnalysisRulePr private final FlowEngine validationThreadPool; private final FlowEngine flowAnalysisThreadPool; private final ValidationTrigger validationTrigger; + private final ConnectorValidationTrigger connectorValidationTrigger; private final ReloadComponent reloadComponent; private final VerifiableComponentFactory verifiableComponentFactory; private final ProvenanceAuthorizableFactory provenanceAuthorizableFactory; @@ -339,6 +362,7 @@ public class FlowController implements ReportingTaskProvider, FlowAnalysisRulePr private final RingBufferGarbageCollectionLog gcLog; private final Optional longRunningTaskMonitorThreadPool; + /** * true if controller is configured to operate in a clustered environment */ @@ -413,7 +437,8 @@ public static FlowController createStandaloneInstance( final ExtensionDiscoveringManager extensionManager, final StatusHistoryRepository statusHistoryRepository, final RuleViolationsManager ruleViolationsManager, - final StateManagerProvider stateManagerProvider + final StateManagerProvider stateManagerProvider, + final ConnectorRequestReplicator connectorRequestReplicator ) { return new FlowController( @@ -434,7 +459,8 @@ public static FlowController createStandaloneInstance( null, statusHistoryRepository, ruleViolationsManager, - stateManagerProvider + stateManagerProvider, + connectorRequestReplicator ); } @@ -455,7 +481,8 @@ public static FlowController createClusteredInstance( final RevisionManager revisionManager, final StatusHistoryRepository statusHistoryRepository, final RuleViolationsManager ruleViolationsManager, - final StateManagerProvider stateManagerProvider + final StateManagerProvider stateManagerProvider, + final ConnectorRequestReplicator connectorRequestReplicator ) { return new FlowController( @@ -476,7 +503,8 @@ public static FlowController createClusteredInstance( revisionManager, statusHistoryRepository, ruleViolationsManager, - stateManagerProvider + stateManagerProvider, + connectorRequestReplicator ); } @@ -498,7 +526,8 @@ private FlowController( final RevisionManager revisionManager, final StatusHistoryRepository statusHistoryRepository, final RuleViolationsManager ruleViolationsManager, - final StateManagerProvider stateManagerProvider + final StateManagerProvider stateManagerProvider, + final ConnectorRequestReplicator connectorRequestReplicator ) { maxTimerDrivenThreads = new AtomicInteger(10); @@ -567,21 +596,26 @@ private FlowController( } lifecycleStateManager = new StandardLifecycleStateManager(); + reloadComponent = new StandardReloadComponent(this); processScheduler = new StandardProcessScheduler(timerDrivenEngineRef.get(), this, stateManagerProvider, this.nifiProperties, lifecycleStateManager); parameterContextManager = new StandardParameterContextManager(); final long maxAppendableBytes = getMaxAppendableBytes(); - repositoryContextFactory = new RepositoryContextFactory( - contentRepository, - flowFileRepository, - flowFileEventRepository, - counterRepositoryRef.get(), - getComponentMetricReporter(), - provenanceRepository, - stateManagerProvider, - maxAppendableBytes + repositoryContextFactory = new RepositoryContextFactory(contentRepository, flowFileRepository, flowFileEventRepository, + counterRepositoryRef.get(), componentMetricReporter, provenanceRepository, stateManagerProvider, maxAppendableBytes); + + assetManager = createAssetManager( + nifiProperties, + NiFiProperties.ASSET_MANAGER_IMPLEMENTATION, + NiFiProperties.ASSET_MANAGER_PREFIX, + DEFAULT_ASSET_MANAGER_IMPLEMENTATION + ); + connectorAssetManager = createAssetManager( + nifiProperties, + NiFiProperties.CONNECTOR_ASSET_MANAGER_IMPLEMENTATION, + NiFiProperties.CONNECTOR_ASSET_MANAGER_PREFIX, + DEFAULT_CONNECTOR_ASSET_MANAGER_IMPLEMENTATION ); - assetManager = createAssetManager(nifiProperties); this.flowAnalysisThreadPool = new FlowEngine(1, "Background Flow Analysis", true); if (ruleViolationsManager != null) { @@ -602,10 +636,17 @@ private FlowController( parameterContextManager ); + connectorRequestReplicator.setFlowManager(flowManager); + controllerServiceProvider = new StandardControllerServiceProvider(processScheduler, bulletinRepository, flowManager, extensionManager); - controllerServiceResolver = new StandardControllerServiceResolver(authorizer, flowManager, new NiFiRegistryFlowMapper(extensionManager), + controllerServiceResolver = new StandardControllerServiceResolver(authorizer, flowManager, new VersionedComponentFlowMapper(extensionManager), controllerServiceProvider, new StandardControllerServiceApiLookup(extensionManager)); + final SecretsManager secretsManager = createSecretsManager(nifiProperties, extensionManager, flowManager); + final ConnectorConfigurationProvider connectorConfigurationProvider = createConnectorConfigurationProvider(nifiProperties, extensionManager, connectorAssetManager); + connectorRepository = createConnectorRepository(nifiProperties, extensionManager, flowManager, connectorAssetManager, secretsManager, this, connectorRequestReplicator, + connectorConfigurationProvider); + final PythonBridge rawPythonBridge = createPythonBridge(nifiProperties, controllerServiceProvider); final ClassLoader pythonBridgeClassLoader = rawPythonBridge.getClass().getClassLoader(); final PythonBridge classloaderAwareBridge = new ClassLoaderAwarePythonBridge(rawPythonBridge, pythonBridgeClassLoader); @@ -658,7 +699,6 @@ private FlowController( this.heartbeatDelaySeconds = (int) FormatUtils.getTimeDuration(nifiProperties.getNodeHeartbeatInterval(), TimeUnit.SECONDS); this.snippetManager = new SnippetManager(); - this.reloadComponent = new StandardReloadComponent(this); this.verifiableComponentFactory = new StandardVerifiableComponentFactory(this, this.nifiProperties); final ProcessGroup rootGroup = flowManager.createProcessGroup(ComponentIdGenerator.generateId().toString()); @@ -668,6 +708,7 @@ private FlowController( this.validationThreadPool = new FlowEngine(5, "Validate Components", true); this.validationTrigger = new StandardValidationTrigger(validationThreadPool, this::isInitialized); + this.connectorValidationTrigger = new StandardConnectorValidationTrigger(validationThreadPool, this::isInitialized); if (remoteInputSocketPort == null) { LOG.info("Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set"); @@ -858,7 +899,7 @@ public Resource getResource() { return ResourceFactory.getControllerResource(); } - private static FlowFileRepository createFlowFileRepository(final NiFiProperties properties, final ExtensionManager extensionManager, final ResourceClaimManager contentClaimManager) { + private static FlowFileRepository createFlowFileRepository(final NiFiProperties properties, final ExtensionManager extensionManager, final ResourceClaimManager resourceClaimManager) { final String implementationClassName = properties.getProperty(NiFiProperties.FLOWFILE_REPOSITORY_IMPLEMENTATION, DEFAULT_FLOWFILE_REPO_IMPLEMENTATION); if (implementationClassName == null) { throw new RuntimeException("Cannot create FlowFile Repository because the NiFi Properties is missing the following property: " @@ -868,7 +909,7 @@ private static FlowFileRepository createFlowFileRepository(final NiFiProperties try { final FlowFileRepository created = NarThreadContextClassLoader.createInstance(extensionManager, implementationClassName, FlowFileRepository.class, properties); synchronized (created) { - created.initialize(contentClaimManager); + created.initialize(resourceClaimManager); } return created; @@ -877,6 +918,167 @@ private static FlowFileRepository createFlowFileRepository(final NiFiProperties } } + private static ConnectorRepository createConnectorRepository(final NiFiProperties properties, final ExtensionDiscoveringManager extensionManager, final FlowManager flowManager, + final AssetManager assetManager, final SecretsManager secretsManager, final NodeTypeProvider nodeTypeProvider, final ConnectorRequestReplicator requestReplicator, + final ConnectorConfigurationProvider connectorConfigurationProvider) { + + final String implementationClassName = properties.getProperty(NiFiProperties.CONNECTOR_REPOSITORY_IMPLEMENTATION, DEFAULT_CONNECTOR_REPOSITORY_IMPLEMENTATION); + + try { + // Discover implementations of Connector Repository. This is not done at startup because the ConnectorRepository class is not + // provided in the list of standard extension points. This is due to the fact that ConnectorRepository lives in the nifi-framework-core-api, and + // does not make sense to refactor it into some other module due to its dependencies, simply to allow it to be discovered at startup. + final Set> additionalExtensionTypes = Set.of(ConnectorRepository.class, SecretsManager.class, ConnectorConfigurationProvider.class); + extensionManager.discoverExtensions(extensionManager.getAllBundles(), additionalExtensionTypes, true); + final ConnectorRepository created = NarThreadContextClassLoader.createInstance(extensionManager, implementationClassName, ConnectorRepository.class, properties); + + final ConnectorRepositoryInitializationContext initializationContext = new StandardConnectorRepoInitializationContext( + flowManager, + extensionManager, + secretsManager, + assetManager, + nodeTypeProvider, + requestReplicator, + connectorConfigurationProvider + ); + + synchronized (created) { + // Ensure that any NAR dependencies are available when we initialize the ConnectorRepository + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, created.getClass(), "connector-repository")) { + created.initialize(initializationContext); + } + } + + LOG.info("Created Connector Repository of type {}", created.getClass().getSimpleName()); + + return created; + } catch (final Exception e) { + throw new RuntimeException(e); + } + } + + private static SecretsManager createSecretsManager(final NiFiProperties properties, final ExtensionDiscoveringManager extensionManager, final FlowManager flowManager) { + final String implementationClassName = properties.getProperty(NiFiProperties.SECRETS_MANAGER_IMPLEMENTATION, DEFAULT_SECRETS_MANAGER_IMPLEMENTATION); + + try { + // Discover implementations of Secrets Manager. This is not done at startup because the SecretsManager class is not + // provided in the list of standard extension points. This is due to the fact that SecretsManager lives in the nifi-framework-core-api, and + // does not make sense to refactor it into some other module due to its dependencies, simply to allow it to be discovered at startup. + extensionManager.discoverExtensions(extensionManager.getAllBundles(), Set.of(SecretsManager.class), true); + final SecretsManager created = NarThreadContextClassLoader.createInstance(extensionManager, implementationClassName, SecretsManager.class, properties); + + final SecretsManagerInitializationContext initializationContext = new StandardSecretsManagerInitializationContext(flowManager); + + synchronized (created) { + // Ensure that any NAR dependencies are available when we initialize the ConnectorRepository + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, created.getClass(), "secrets-manager")) { + created.initialize(initializationContext); + } + } + + LOG.info("Created Secrets Manager of type {}", created.getClass().getSimpleName()); + + return created; + } catch (final Exception e) { + throw new RuntimeException(e); + } + } + + private static ConnectorConfigurationProvider createConnectorConfigurationProvider(final NiFiProperties properties, final ExtensionDiscoveringManager extensionManager, + final AssetManager connectorAssetManager) { + final String implementationClassName = properties.getProperty(NiFiProperties.CONNECTOR_CONFIGURATION_PROVIDER_IMPLEMENTATION); + if (implementationClassName == null || implementationClassName.isBlank()) { + LOG.info("No Connector Configuration Provider implementation configured; external connector configuration management is disabled"); + return null; + } + + try { + extensionManager.discoverExtensions(extensionManager.getAllBundles(), Set.of(ConnectorConfigurationProvider.class), true); + final ConnectorConfigurationProvider created = NarThreadContextClassLoader.createInstance( + extensionManager, implementationClassName, ConnectorConfigurationProvider.class, properties); + + final Map initializationProperties = properties.getPropertiesWithPrefix(NiFiProperties.CONNECTOR_CONFIGURATION_PROVIDER_PROPERTIES_PREFIX) + .entrySet().stream() + .collect(Collectors.toMap( + entry -> entry.getKey().substring(NiFiProperties.CONNECTOR_CONFIGURATION_PROVIDER_PROPERTIES_PREFIX.length()), + Map.Entry::getValue + )); + + final ConnectorConfigurationProviderInitializationContext initializationContext = + new StandardConnectorConfigurationProviderInitializationContext(initializationProperties, connectorAssetManager); + + synchronized (created) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, created.getClass(), "connector-configuration-provider")) { + created.initialize(initializationContext); + } + } + + LOG.info("Created Connector Configuration Provider of type {}", created.getClass().getSimpleName()); + + return created; + } catch (final Exception e) { + throw new RuntimeException("Failed to create Connector Configuration Provider", e); + } + } + + public ConnectorRepository getConnectorRepository() { + return connectorRepository; + } + + /** + * Finds a Connection by ID, searching both the root process group hierarchy + * and all connector-managed process groups. + * + * @param connectionId the connection identifier + * @return the Connection, or null if not found + */ + public Connection findConnectionIncludingConnectorManaged(final String connectionId) { + final Connection connection = flowManager.getRootGroup().findConnection(connectionId); + if (connection != null) { + return connection; + } + + for (final ConnectorNode connector : connectorRepository.getConnectors()) { + final FrameworkFlowContext flowContext = connector.getActiveFlowContext(); + if (flowContext != null) { + final ProcessGroup managedGroup = flowContext.getManagedProcessGroup(); + final Connection managedConnection = managedGroup.findConnection(connectionId); + if (managedConnection != null) { + return managedConnection; + } + } + } + + return null; + } + + /** + * Finds a RemoteGroupPort by ID, searching both the root process group hierarchy + * and all connector-managed process groups. + * + * @param remoteGroupPortId the remote group port identifier + * @return the RemoteGroupPort, or null if not found + */ + public RemoteGroupPort findRemoteGroupPortIncludingConnectorManaged(final String remoteGroupPortId) { + final RemoteGroupPort remoteGroupPort = flowManager.getRootGroup().findRemoteGroupPort(remoteGroupPortId); + if (remoteGroupPort != null) { + return remoteGroupPort; + } + + for (final ConnectorNode connector : connectorRepository.getConnectors()) { + final FrameworkFlowContext flowContext = connector.getActiveFlowContext(); + if (flowContext != null) { + final ProcessGroup managedGroup = flowContext.getManagedProcessGroup(); + final RemoteGroupPort managedPort = managedGroup.findRemoteGroupPort(remoteGroupPortId); + if (managedPort != null) { + return managedPort; + } + } + } + + return null; + } + private PythonBridge createPythonBridge(final NiFiProperties nifiProperties, final ControllerServiceProvider serviceProvider) { final String pythonCommand = nifiProperties.getProperty(NiFiProperties.PYTHON_COMMAND); if (pythonCommand == null) { @@ -1192,7 +1394,7 @@ public void onFlowInitialized(final boolean startDelayedComponents) { Supplier rootProcessGroupSupplier = () -> { ProcessGroup rootProcessGroup = getFlowManager().getRootGroup(); - NiFiRegistryFlowMapper mapper = FlowAnalysisUtil.createMapper(getExtensionManager()); + VersionedComponentFlowMapper mapper = FlowAnalysisUtil.createMapper(getExtensionManager()); InstantiatedVersionedProcessGroup versionedRootProcessGroup = mapper.mapNonVersionedProcessGroup( rootProcessGroup, @@ -1390,14 +1592,14 @@ private ContentRepository createContentRepository(final NiFiProperties propertie } } - private AssetManager createAssetManager(final NiFiProperties properties) { - final String implementationClassName = properties.getProperty(NiFiProperties.ASSET_MANAGER_IMPLEMENTATION, DEFAULT_ASSET_MANAGER_IMPLEMENTATION); + private AssetManager createAssetManager(final NiFiProperties properties, final String implementationClassProperty, final String propertyPrefix, final String defaultImplementationClass) { + final String implementationClassName = properties.getProperty(implementationClassProperty, defaultImplementationClass); try { final AssetManager assetManager = NarThreadContextClassLoader.createInstance(extensionManager, implementationClassName, AssetManager.class, properties); final AssetReferenceLookup assetReferenceLookup = new StandardAssetReferenceLookup(parameterContextManager); - final Map relevantNiFiProperties = properties.getPropertiesWithPrefix(NiFiProperties.ASSET_MANAGER_PREFIX); - final int prefixLength = NiFiProperties.ASSET_MANAGER_PREFIX.length(); + final Map relevantNiFiProperties = properties.getPropertiesWithPrefix(propertyPrefix); + final int prefixLength = propertyPrefix.length(); final Map assetManagerProperties = relevantNiFiProperties.entrySet().stream() .collect(Collectors.toMap(entry -> entry.getKey().substring(prefixLength), Map.Entry::getValue)); @@ -1415,9 +1617,16 @@ public void initialize(final AssetManagerInitializationContext context) { } @Override - public Asset createAsset(final String parameterContextId, final String assetName, final InputStream contents) throws IOException { + public Asset createAsset(final String ownerId, final String assetName, final InputStream contents) throws IOException { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(assetManagerClassLoader)) { + return assetManager.createAsset(ownerId, assetName, contents); + } + } + + @Override + public Asset saveAsset(final String ownerId, final String assetId, final String assetName, final InputStream contents) throws IOException { try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(assetManagerClassLoader)) { - return assetManager.createAsset(parameterContextId, assetName, contents); + return assetManager.saveAsset(ownerId, assetId, assetName, contents); } } @@ -1429,16 +1638,16 @@ public Optional getAsset(final String id) { } @Override - public List getAssets(final String parameterContextId) { + public List getAssets(final String ownerId) { try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(assetManagerClassLoader)) { - return assetManager.getAssets(parameterContextId); + return assetManager.getAssets(ownerId); } } @Override - public Asset createMissingAsset(final String parameterContextId, final String assetName) { + public Asset createMissingAsset(final String ownerId, final String assetName) { try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(assetManagerClassLoader)) { - return assetManager.createMissingAsset(parameterContextId, assetName); + return assetManager.createMissingAsset(ownerId, assetName); } } @@ -1462,6 +1671,10 @@ public AssetManager getAssetManager() { return assetManager; } + public AssetManager getConnectorAssetManager() { + return connectorAssetManager; + } + private ProvenanceRepository createProvenanceRepository(final NiFiProperties properties) { final String implementationClassName = properties.getProperty(NiFiProperties.PROVENANCE_REPO_IMPLEMENTATION_CLASS, DEFAULT_PROVENANCE_REPO_IMPLEMENTATION); if (StringUtils.isBlank(implementationClassName)) { @@ -1494,6 +1707,10 @@ public ValidationTrigger getValidationTrigger() { return validationTrigger; } + public ConnectorValidationTrigger getConnectorValidationTrigger() { + return connectorValidationTrigger; + } + public PropertyEncryptor getEncryptor() { return encryptor; } @@ -1530,6 +1747,7 @@ public BulletinRepository getBulletinRepository() { public LifecycleStateManager getLifecycleStateManager() { return lifecycleStateManager; } + public SnippetManager getSnippetManager() { return snippetManager; } @@ -2075,7 +2293,7 @@ private Set findAllConnections(final VersionedProcessGroup * @return the process group or null if not group is found */ private ProcessGroup lookupGroup(final String id) { - final ProcessGroup group = flowManager.getGroup(id); + final ProcessGroup group = flowManager.getGroup(id, null); if (group == null) { throw new IllegalStateException("No Group with ID " + id + " exists"); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java index 9cdab7879a4a..fc212ad88e41 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java @@ -122,7 +122,8 @@ public class StandardFlowService implements FlowService, ProtocolHandler { private final ClusterCoordinator clusterCoordinator; private final RevisionManager revisionManager; private final NarManager narManager; - private final AssetSynchronizer assetSynchronizer; + private final AssetSynchronizer parameterContextAssetSynchronizer; + private final AssetSynchronizer connectorAssetSynchronizer; private volatile SaveReportingTask saveReportingTask; /** @@ -153,10 +154,12 @@ public static StandardFlowService createStandaloneInstance( final NiFiProperties nifiProperties, final RevisionManager revisionManager, final NarManager narManager, - final AssetSynchronizer assetSynchronizer, + final AssetSynchronizer parameterContextAssetSynchronizer, + final AssetSynchronizer connectorAssetSynchronizer, final Authorizer authorizer) throws IOException { - return new StandardFlowService(controller, nifiProperties, null, false, null, revisionManager, narManager, assetSynchronizer, authorizer); + return new StandardFlowService(controller, nifiProperties, null, false, null, revisionManager, narManager, + parameterContextAssetSynchronizer, connectorAssetSynchronizer, authorizer); } public static StandardFlowService createClusteredInstance( @@ -166,10 +169,12 @@ public static StandardFlowService createClusteredInstance( final ClusterCoordinator coordinator, final RevisionManager revisionManager, final NarManager narManager, - final AssetSynchronizer assetSynchronizer, + final AssetSynchronizer parameterContextAssetSynchronizer, + final AssetSynchronizer connectorAssetSynchronizer, final Authorizer authorizer) throws IOException { - return new StandardFlowService(controller, nifiProperties, senderListener, true, coordinator, revisionManager, narManager, assetSynchronizer, authorizer); + return new StandardFlowService(controller, nifiProperties, senderListener, true, coordinator, revisionManager, + narManager, parameterContextAssetSynchronizer, connectorAssetSynchronizer, authorizer); } private StandardFlowService( @@ -180,7 +185,8 @@ private StandardFlowService( final ClusterCoordinator clusterCoordinator, final RevisionManager revisionManager, final NarManager narManager, - final AssetSynchronizer assetSynchronizer, + final AssetSynchronizer parameterContextAssetSynchronizer, + final AssetSynchronizer connectorAssetSynchronizer, final Authorizer authorizer) throws IOException { this.nifiProperties = nifiProperties; @@ -196,7 +202,8 @@ private StandardFlowService( } this.revisionManager = revisionManager; this.narManager = narManager; - this.assetSynchronizer = assetSynchronizer; + this.parameterContextAssetSynchronizer = parameterContextAssetSynchronizer; + this.connectorAssetSynchronizer = connectorAssetSynchronizer; this.authorizer = authorizer; if (configuredForClustering) { @@ -926,8 +933,13 @@ private void loadFromConnectionResponse(final ConnectionResponse response) throw // load new controller state loadFromBytes(dataFlow, true, BundleUpdateStrategy.USE_SPECIFIED_OR_COMPATIBLE_OR_GHOST); - // sync assets after loading the flow so that parameter contexts exist first - assetSynchronizer.synchronize(); + // sync assets after loading the flow so that parameter contexts and connectors exist first + if (parameterContextAssetSynchronizer != null) { + parameterContextAssetSynchronizer.synchronize(); + } + if (connectorAssetSynchronizer != null) { + connectorAssetSynchronizer.synchronize(); + } // set node ID on controller before we start heartbeating because heartbeat needs node ID clusterCoordinator.setLocalNodeIdentifier(nodeId); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/FlowControllerFlowContextFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/FlowControllerFlowContextFactory.java new file mode 100644 index 000000000000..dbca31b7e327 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/FlowControllerFlowContextFactory.java @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller.flow; + +import org.apache.nifi.components.connector.FlowContextFactory; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.MutableConnectorConfigurationContext; +import org.apache.nifi.components.connector.ParameterContextFacadeFactory; +import org.apache.nifi.components.connector.ProcessGroupFacadeFactory; +import org.apache.nifi.components.connector.ProcessGroupFactory; +import org.apache.nifi.components.connector.StandardFlowContext; +import org.apache.nifi.components.connector.components.FlowContextType; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.connector.components.ParameterValue; +import org.apache.nifi.components.connector.facades.standalone.StandaloneParameterContextFacade; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; +import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; +import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +public class FlowControllerFlowContextFactory implements FlowContextFactory { + private final FlowController flowController; + private final ProcessGroup activeManagedProcessGroup; + private final MutableConnectorConfigurationContext activeConfigurationContext; + private final ParameterContextFacadeFactory parameterContextFacadeFactory; + private final ProcessGroupFacadeFactory processGroupFacadeFactory; + private final ProcessGroupFactory processGroupFactory; + + public FlowControllerFlowContextFactory(final FlowController flowController, final ProcessGroup activeManagedProcessGroup, + final MutableConnectorConfigurationContext activeConfigurationContext, final ProcessGroupFactory processGroupFactory) { + + this.flowController = flowController; + this.activeManagedProcessGroup = activeManagedProcessGroup; + this.activeConfigurationContext = activeConfigurationContext; + this.processGroupFactory = processGroupFactory; + + this.processGroupFacadeFactory = new FlowControllerProcessGroupFacadeFactory(flowController); + this.parameterContextFacadeFactory = processGroup -> new StandaloneParameterContextFacade(flowController, processGroup); + } + + @Override + public FrameworkFlowContext createActiveFlowContext(final String connectorId, final ComponentLog connectorLogger, final Bundle bundle) { + return new StandardFlowContext(activeManagedProcessGroup, activeConfigurationContext, processGroupFacadeFactory, + parameterContextFacadeFactory, connectorLogger, FlowContextType.ACTIVE, bundle); + } + + @Override + public FrameworkFlowContext createWorkingFlowContext(final String connectorId, final ComponentLog connectorLogger, + final MutableConnectorConfigurationContext activeConfigurationContext, final Bundle bundle) { + + final String workingGroupId = UUID.nameUUIDFromBytes((connectorId + "-working").getBytes(StandardCharsets.UTF_8)).toString(); + final ProcessGroup processGroup = processGroupFactory.create(workingGroupId); + copyGroupContents(activeManagedProcessGroup, processGroup, connectorId + "-working-context"); + + final MutableConnectorConfigurationContext workingConfigurationContext = activeConfigurationContext.clone(); + + return new StandardFlowContext(processGroup, workingConfigurationContext, processGroupFacadeFactory, + parameterContextFacadeFactory, connectorLogger, FlowContextType.WORKING, bundle); + } + + private void copyGroupContents(final ProcessGroup sourceGroup, final ProcessGroup destinationGroup, final String componentIdSeed) { + final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder() + .mapSensitiveConfiguration(true) + .mapPropertyDescriptors(true) + .stateLookup(VersionedComponentStateLookup.ENABLED_OR_DISABLED) + .sensitiveValueEncryptor(value -> value) + .componentIdLookup(ComponentIdLookup.VERSIONED_OR_GENERATE) + .mapInstanceIdentifiers(true) + .mapControllerServiceReferencesToVersionedId(true) + .mapFlowRegistryClientId(true) + .mapAssetReferences(true) + .build(); + + final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(flowController.getExtensionManager(), flowMappingOptions); + final InstantiatedVersionedProcessGroup versionedGroup = flowMapper.mapProcessGroup(sourceGroup, flowController.getControllerServiceProvider(), + flowController.getFlowManager(), true); + + final String contextName = sourceGroup.getParameterContext().getName(); + + final VersionedExternalFlow externalFlowWithoutParameterContext = new VersionedExternalFlow(); + externalFlowWithoutParameterContext.setFlowContents(versionedGroup); + externalFlowWithoutParameterContext.setParameterContexts(Map.of()); + + final String duplicateContextId = UUID.nameUUIDFromBytes((destinationGroup.getIdentifier() + "-param-context").getBytes(StandardCharsets.UTF_8)).toString(); + final ParameterContext sourceContext = sourceGroup.getParameterContext(); + final ParameterContext duplicateParameterContext = flowController.getFlowManager().createEmptyParameterContext( + duplicateContextId, contextName, sourceContext.getDescription(), destinationGroup); + + destinationGroup.setParameterContext(duplicateParameterContext); + destinationGroup.updateFlow(externalFlowWithoutParameterContext, componentIdSeed, false, true, true); + + final ParameterContextFacade contextFacade = new StandaloneParameterContextFacade(flowController, destinationGroup); + final List parameterValues = createParameterValues(sourceContext); + contextFacade.updateParameters(parameterValues); + } + + private List createParameterValues(final ParameterContext context) { + final List parameterValues = new ArrayList<>(); + for (final Parameter parameter : context.getParameters().values()) { + final ParameterValue.Builder parameterValueBuilder = new ParameterValue.Builder() + .name(parameter.getDescriptor().getName()) + .sensitive(parameter.getDescriptor().isSensitive()) + .value(parameter.getValue()); + + parameter.getReferencedAssets().forEach(parameterValueBuilder::addReferencedAsset); + parameterValues.add(parameterValueBuilder.build()); + } + + return parameterValues; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/FlowControllerProcessGroupFacadeFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/FlowControllerProcessGroupFacadeFactory.java new file mode 100644 index 000000000000..53bbe05b9ff2 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/FlowControllerProcessGroupFacadeFactory.java @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller.flow; + +import org.apache.nifi.components.connector.ProcessGroupFacadeFactory; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.components.connector.facades.standalone.ComponentContextProvider; +import org.apache.nifi.components.connector.facades.standalone.StandaloneProcessGroupFacade; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.ProcessScheduler; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; +import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; + +public class FlowControllerProcessGroupFacadeFactory implements ProcessGroupFacadeFactory { + private final FlowController flowController; + + public FlowControllerProcessGroupFacadeFactory(final FlowController flowController) { + this.flowController = flowController; + } + + @Override + public ProcessGroupFacade create(final ProcessGroup processGroup, final ComponentLog connectorLogger) { + final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder() + .mapSensitiveConfiguration(true) + .mapPropertyDescriptors(true) + .stateLookup(VersionedComponentStateLookup.IDENTITY_LOOKUP) + .sensitiveValueEncryptor(value -> value) + .componentIdLookup(ComponentIdLookup.VERSIONED_OR_GENERATE) + .mapInstanceIdentifiers(true) + .mapControllerServiceReferencesToVersionedId(true) + .mapFlowRegistryClientId(true) + .mapAssetReferences(true) + .build(); + + final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(flowController.getExtensionManager(), flowMappingOptions); + + final VersionedProcessGroup versionedManagedGroup = flowMapper.mapProcessGroup(processGroup, flowController.getControllerServiceProvider(), + flowController.getFlowManager(), true); + final ComponentContextProvider componentContextProvider = new StandardComponentContextProvider(flowController); + final ParameterContext parameterContext = processGroup.getParameterContext(); + final ProcessScheduler processScheduler = flowController.getProcessScheduler(); + final ExtensionManager extensionManager = flowController.getExtensionManager(); + + return new StandaloneProcessGroupFacade(processGroup, versionedManagedGroup, + processScheduler, parameterContext, flowController.getControllerServiceProvider(), + componentContextProvider, connectorLogger, extensionManager, flowController.getAssetManager()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardComponentContextProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardComponentContextProvider.java new file mode 100644 index 000000000000..89e1c215a7a0 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardComponentContextProvider.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller.flow; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.connector.facades.standalone.ComponentContextProvider; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.controller.ComponentNode; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.lifecycle.TaskTermination; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.controller.service.StandardConfigurationContext; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.parameter.ParameterLookup; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.StandardProcessContext; + +import java.util.HashMap; +import java.util.Map; +import java.util.stream.Collectors; + +public class StandardComponentContextProvider implements ComponentContextProvider { + private final FlowController flowController; + + public StandardComponentContextProvider(final FlowController flowController) { + this.flowController = flowController; + } + + @Override + public ProcessContext createProcessContext(final ProcessorNode processorNode, final ParameterLookup parameterLookup) { + return createProcessContext(processorNode, Map.of(), parameterLookup); + } + + @Override + public ProcessContext createProcessContext(final ProcessorNode processorNode, final Map propertiesOverride, final ParameterLookup parameterLookup) { + final StateManager stateManager = flowController.getStateManagerProvider().getStateManager(processorNode.getIdentifier()); + final TaskTermination taskTermination = () -> false; + final Map serviceReferencedProperties = resolveServiceReferences(processorNode, processorNode.getProcessGroup(), propertiesOverride); + return new StandardProcessContext(processorNode, serviceReferencedProperties, null, parameterLookup, + flowController.getControllerServiceProvider(), stateManager, taskTermination, flowController); + } + + @Override + public ValidationContext createValidationContext(final ProcessorNode processorNode, final Map properties, final ParameterLookup parameterLookup) { + final Map serviceReferencedProperties = resolveServiceReferences(processorNode, processorNode.getProcessGroup(), properties); + return processorNode.createValidationContext(serviceReferencedProperties, processorNode.getAnnotationData(), parameterLookup, true); + } + + private Map resolveServiceReferences(final ComponentNode processorNode, final ProcessGroup processGroup, final Map propertiesOverride) { + final Map versionedToInstanceIds = processGroup.findAllControllerServices().stream() + .filter(cs -> cs.getVersionedComponentId().isPresent()) + .collect(Collectors.toMap(cs -> cs.getVersionedComponentId().get(), ComponentNode::getIdentifier)); + + final Map resolved = new HashMap<>(propertiesOverride); + for (final PropertyDescriptor descriptor : processorNode.getPropertyDescriptors()) { + if (descriptor.getControllerServiceDefinition() == null) { + continue; + } + + final String overriddenValue = propertiesOverride.get(descriptor.getName()); + final String currentValue = overriddenValue == null ? processorNode.getRawPropertyValue(descriptor) : overriddenValue; + final String instanceId = versionedToInstanceIds.get(currentValue); + if (instanceId != null) { + resolved.put(descriptor.getName(), instanceId); + } + } + + return resolved; + } + + @Override + public ConfigurationContext createConfigurationContext(final ControllerServiceNode serviceNode, final ParameterLookup parameterLookup) { + return createConfigurationContext(serviceNode, Map.of(), parameterLookup); + } + + @Override + public ConfigurationContext createConfigurationContext(final ControllerServiceNode serviceNode, final Map propertiesOverride, final ParameterLookup parameterLookup) { + final Map resolvedProperties = resolveServiceReferences(serviceNode, serviceNode.getProcessGroup(), propertiesOverride); + return new StandardConfigurationContext(serviceNode, resolvedProperties, null, parameterLookup, flowController.getControllerServiceProvider(), null); + } + + @Override + public ValidationContext createValidationContext(final ControllerServiceNode serviceNode, final Map properties, final ParameterLookup parameterLookup) { + final Map resolvedProperties = resolveServiceReferences(serviceNode, serviceNode.getProcessGroup(), properties); + return serviceNode.createValidationContext(resolvedProperties, serviceNode.getAnnotationData(), parameterLookup, true); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardFlowManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardFlowManager.java index e6ba2c0d4d26..144f3cde8c56 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardFlowManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardFlowManager.java @@ -27,6 +27,15 @@ import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.components.ConfigurableComponent; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.ComponentBundleLookup; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorStateTransition; +import org.apache.nifi.components.connector.FlowContextFactory; +import org.apache.nifi.components.connector.ProcessGroupFactory; +import org.apache.nifi.components.connector.StandardComponentBundleLookup; +import org.apache.nifi.components.connector.StandardConnectorConfigurationContext; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; @@ -56,12 +65,15 @@ import org.apache.nifi.controller.service.StandardConfigurationContext; import org.apache.nifi.deprecation.log.DeprecationLogger; import org.apache.nifi.deprecation.log.DeprecationLoggerFactory; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameterContext; import org.apache.nifi.flowanalysis.FlowAnalysisRule; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.groups.RemoteProcessGroup; import org.apache.nifi.groups.StandardProcessGroup; import org.apache.nifi.groups.StatelessGroupNodeFactory; +import org.apache.nifi.logging.ConnectorLogObserver; import org.apache.nifi.logging.ControllerServiceLogObserver; import org.apache.nifi.logging.FlowAnalysisRuleLogObserver; import org.apache.nifi.logging.FlowRegistryClientLogObserver; @@ -73,10 +85,18 @@ import org.apache.nifi.logging.ReportingTaskLogObserver; import org.apache.nifi.nar.ExtensionManager; import org.apache.nifi.nar.NarCloseable; +import org.apache.nifi.parameter.ParameterContext; import org.apache.nifi.parameter.ParameterContextManager; import org.apache.nifi.parameter.ParameterProvider; +import org.apache.nifi.parameter.ParameterReferenceManager; +import org.apache.nifi.parameter.StandardParameterReferenceManager; import org.apache.nifi.processor.Processor; import org.apache.nifi.registry.flow.FlowRegistryClientNode; +import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; +import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; +import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; import org.apache.nifi.remote.PublicPort; import org.apache.nifi.remote.StandardPublicPort; import org.apache.nifi.remote.StandardRemoteProcessGroup; @@ -92,6 +112,7 @@ import java.lang.reflect.InvocationTargetException; import java.net.URL; +import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; @@ -100,6 +121,7 @@ import java.util.Map; import java.util.Optional; import java.util.Set; +import java.util.UUID; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.TimeUnit; @@ -279,13 +301,14 @@ public Port createLocalOutputPort(String id, String name) { } @Override - public ProcessGroup createProcessGroup(final String id) { + public ProcessGroup createProcessGroup(final String id, final String connectorId) { final StatelessGroupNodeFactory statelessGroupNodeFactory = new StandardStatelessGroupNodeFactory(flowController, sslContext, flowController.createKerberosConfig(nifiProperties)); final ProcessGroup group = new StandardProcessGroup(requireNonNull(id), flowController.getControllerServiceProvider(), processScheduler, flowController.getEncryptor(), flowController.getExtensionManager(), flowController.getStateManagerProvider(), this, flowController.getReloadComponent(), flowController, nifiProperties, statelessGroupNodeFactory, - flowController.getAssetManager()); + flowController.getAssetManager(), connectorId); + onProcessGroupAdded(group); return group; @@ -718,6 +741,126 @@ public void removeRootControllerService(final ControllerServiceNode service) { logger.info("{} removed from Flow Controller", service); } + @Override + public ConnectorNode createConnector(final String type, final String id, final BundleCoordinate coordinate, final boolean firstTimeAdded, final boolean registerLogObserver) { + + requireNonNull(type, "Connector Type"); + requireNonNull(id, "Connector ID"); + requireNonNull(coordinate, "Bundle Coordinate"); + + final ExtensionManager extensionManager = flowController.getExtensionManager(); + + final String managedGroupId = UUID.nameUUIDFromBytes((id + "-root").getBytes(StandardCharsets.UTF_8)).toString(); + final ProcessGroup managedRootGroup = createProcessGroup(managedGroupId, id); + + final String paramContextId = UUID.nameUUIDFromBytes((id + "-parameter-context").getBytes(StandardCharsets.UTF_8)).toString(); + final String paramContextName = "Connector " + id + " Parameter Context"; + final String parameterContextDescription = "Implicit Parameter Context for Connector " + id; + final ParameterReferenceManager referenceManager = new StandardParameterReferenceManager(() -> managedRootGroup); + final ParameterContext managedParameterContext = createParameterContext(paramContextId, paramContextName, + parameterContextDescription, Collections.emptyMap(), Collections.emptyList(), null, referenceManager, false); + managedRootGroup.setParameterContext(managedParameterContext); + + final ConnectorRepository connectorRepository = flowController.getConnectorRepository(); + final ConnectorStateTransition stateTransition = connectorRepository.createStateTransition(type, id); + final StandardConnectorConfigurationContext activeConfigurationContext = new StandardConnectorConfigurationContext( + flowController.getConnectorAssetManager(), flowController.getConnectorRepository().getSecretsManager()); + + final ProcessGroupFactory processGroupFactory = groupId -> createProcessGroup(groupId, id); + final FlowContextFactory flowContextFactory = new FlowControllerFlowContextFactory(flowController, managedRootGroup, activeConfigurationContext, processGroupFactory); + final ComponentBundleLookup componentBundleLookup = new StandardComponentBundleLookup(extensionManager); + + final ConnectorNode connectorNode = new ExtensionBuilder() + .identifier(id) + .type(type) + .bundleCoordinate(coordinate) + .extensionManager(extensionManager) + .managedProcessGroup(managedRootGroup) + .activeConfigurationContext(activeConfigurationContext) + .flowContextFactory(flowContextFactory) + .flowController(flowController) + .connectorStateTransition(stateTransition) + .connectorInitializationContextBuilder(flowController.getConnectorRepository().createInitializationContextBuilder()) + .connectorValidationTrigger(flowController.getConnectorValidationTrigger()) + .componentBundleLookup(componentBundleLookup) + .buildConnector(firstTimeAdded); + + // Establish the Connector as the parent authorizable of the managed root group + managedRootGroup.setExplicitParentAuthorizable(connectorNode); + + // Set up logging for the connector + final LogRepository logRepository = LogRepositoryFactory.getRepository(id); + logRepository.setLogger(connectorNode.getComponentLog()); + if (registerLogObserver) { + logRepository.addObserver(LogLevel.WARN, new ConnectorLogObserver(bulletinRepository, connectorNode)); + } + + if (firstTimeAdded) { + final Connector connector = connectorNode.getConnector(); + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connector.getClass(), connectorNode.getIdentifier())) { + ReflectionUtils.invokeMethodsWithAnnotation(OnAdded.class, connector); + } catch (final Exception e) { + if (registerLogObserver) { + logRepository.removeAllObservers(); + } + throw new ComponentLifeCycleException("Failed to invoke @OnAdded methods of " + connectorNode.getConnector(), e); + } + + flowController.getConnectorRepository().addConnector(connectorNode); + } else { + flowController.getConnectorRepository().restoreConnector(connectorNode); + } + + return connectorNode; + } + + private void copyGroupContents(final ProcessGroup sourceGroup, final ProcessGroup destinationGroup, final String componentIdSeed) { + final FlowMappingOptions flowMappingOptions = new FlowMappingOptions.Builder() + .mapSensitiveConfiguration(true) + .mapPropertyDescriptors(true) + .stateLookup(VersionedComponentStateLookup.ENABLED_OR_DISABLED) + .sensitiveValueEncryptor(value -> value) + .componentIdLookup(ComponentIdLookup.VERSIONED_OR_GENERATE) + .mapInstanceIdentifiers(true) + .mapControllerServiceReferencesToVersionedId(true) + .mapFlowRegistryClientId(true) + .mapAssetReferences(true) + .build(); + + final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(flowController.getExtensionManager(), flowMappingOptions); + final Map parameterContexts = flowMapper.mapParameterContexts(sourceGroup, true, Map.of()); + final InstantiatedVersionedProcessGroup versionedGroup = flowMapper.mapProcessGroup(sourceGroup, flowController.getControllerServiceProvider(), this, true); + final VersionedExternalFlow versionedExternalFlow = new VersionedExternalFlow(); + versionedExternalFlow.setFlowContents(versionedGroup); + versionedExternalFlow.setExternalControllerServices(Map.of()); + versionedExternalFlow.setParameterProviders(Map.of()); + versionedExternalFlow.setParameterContexts(parameterContexts); + + destinationGroup.updateFlow(versionedExternalFlow, componentIdSeed, false, true, true); + } + + private void gatherParameterContexts(final ProcessGroup sourceGroup, final Map parameterContexts) { + final ParameterContext parameterContext = sourceGroup.getParameterContext(); + if (parameterContext != null && !parameterContexts.containsKey(parameterContext.getIdentifier())) { + parameterContexts.put(parameterContext.getIdentifier(), parameterContext); + } + + for (final ProcessGroup childGroup : sourceGroup.getProcessGroups()) { + gatherParameterContexts(childGroup, parameterContexts); + } + } + + + @Override + public List getAllConnectors() { + return flowController.getConnectorRepository().getConnectors(); + } + + @Override + public ConnectorNode getConnector(final String id) { + return flowController.getConnectorRepository().getConnector(id); + } + @Override public ControllerServiceNode createControllerService(final String type, final String id, final BundleCoordinate bundleCoordinate, final Set additionalUrls, final boolean firstTimeAdded, final boolean registerLogObserver, final String classloaderIsolationKey) { diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardStatelessGroupNodeFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardStatelessGroupNodeFactory.java index c5e56cb7bf41..eef4bb0da959 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardStatelessGroupNodeFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flow/StandardStatelessGroupNodeFactory.java @@ -61,7 +61,7 @@ import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; import org.apache.nifi.reporting.BulletinRepository; import org.apache.nifi.stateless.engine.ProcessContextFactory; @@ -186,12 +186,22 @@ public ProcessGroup createStatelessProcessGroup(final ProcessGroup group, final } private VersionedExternalFlow createVersionedExternalFlow(final ProcessGroup group, final FlowMappingOptions flowMappingOptions) { - final NiFiRegistryFlowMapper flowMapper = new NiFiRegistryFlowMapper(flowController.getExtensionManager(), flowMappingOptions); + final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(flowController.getExtensionManager(), flowMappingOptions); final InstantiatedVersionedProcessGroup versionedGroup = flowMapper.mapNonVersionedProcessGroup(group, flowController.getControllerServiceProvider()); - final Map parameterContexts = flowMapper.mapParameterContexts(group, true, new HashMap<>()); final Map externalControllerServiceReferences = Optional.ofNullable(versionedGroup.getExternalControllerServiceReferences()).orElse(Collections.emptyMap()); + // If the Process Group is within a Connector, then we do not want to map Parameter Contexts + // because we do not use the standard management for Parameter Contexts and instead use only an implicit + // Parameter Context for the entire Connector. + final boolean inConnector = group.getConnectorIdentifier().isPresent(); + final Map parameterContexts; + if (inConnector) { + parameterContexts = Collections.emptyMap(); + } else { + parameterContexts = flowMapper.mapParameterContexts(group, true, new HashMap<>()); + } + final VersionedExternalFlow versionedExternalFlow = new VersionedExternalFlow(); versionedExternalFlow.setFlowContents(versionedGroup); versionedExternalFlow.setExternalControllerServices(externalControllerServiceReferences); @@ -299,6 +309,13 @@ public Future> fetch(final Set bundleCoordinates, child.synchronizeFlow(versionedExternalFlow, synchronizationOptions, flowMappingOptions); child.setParent(group); + // If this Process Group is within a Connector, explicitly set the Parameter Context + // on the child Process Group to be the same as the parent group because we want all + // groups within a Connector to share the same Parameter Context. + if (group.getConnectorIdentifier().isPresent()) { + child.setParameterContext(group.getParameterContext()); + } + return child; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisContext.java index 70775f285d6e..d5efbab84c5e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisContext.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisContext.java @@ -24,7 +24,7 @@ import org.apache.nifi.flow.VersionedControllerService; import org.apache.nifi.flowanalysis.FlowAnalysisContext; import org.apache.nifi.nar.ExtensionManager; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import java.util.Collections; import java.util.HashMap; @@ -43,7 +43,7 @@ public VersionedControllerServiceLookup getVersionedControllerServiceLookup() { ControllerServiceProvider controllerServiceProvider = flowController.getControllerServiceProvider(); ExtensionManager extensionManager = flowController.getExtensionManager(); - NiFiRegistryFlowMapper mapper = FlowAnalysisUtil.createMapper(extensionManager); + VersionedComponentFlowMapper mapper = FlowAnalysisUtil.createMapper(extensionManager); ControllerServiceNode controllerServiceNode = controllerServiceProvider.getControllerServiceNode(id); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisRuleNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisRuleNode.java index 2404f48cc5c4..9dda4fa789f2 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisRuleNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/flowanalysis/StandardFlowAnalysisRuleNode.java @@ -22,6 +22,7 @@ import org.apache.nifi.authorization.resource.Authorizable; import org.apache.nifi.authorization.resource.ResourceFactory; import org.apache.nifi.authorization.resource.ResourceType; +import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.validation.ValidationTrigger; import org.apache.nifi.controller.FlowAnalysisRuleNode; @@ -97,7 +98,7 @@ public FlowAnalysisRuleContext getFlowAnalysisRuleContext() { } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java index 6ae3f36bfc4e..dfcfeecfeb04 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java @@ -387,12 +387,16 @@ public long getPenaltyExpirationMillis() { } protected QueueSize drop(final List flowFiles, final String requestor) throws IOException { + return dropWithDetails(flowFiles, "FlowFile Queue emptied by " + requestor); + } + + protected QueueSize dropWithDetails(final List flowFiles, final String details) throws IOException { // Create a Provenance Event and a FlowFile Repository record for each FlowFile final List provenanceEvents = new ArrayList<>(flowFiles.size()); final List flowFileRepoRecords = new ArrayList<>(flowFiles.size()); long dropContentSize = 0L; for (final FlowFileRecord flowFile : flowFiles) { - provenanceEvents.add(createDropProvenanceEvent(flowFile, requestor)); + provenanceEvents.add(createDropProvenanceEvent(flowFile, details)); flowFileRepoRecords.add(createDeleteRepositoryRecord(flowFile)); dropContentSize += flowFile.getSize(); } @@ -402,7 +406,31 @@ protected QueueSize drop(final List flowFiles, final String requ return new QueueSize(flowFiles.size(), dropContentSize); } - private ProvenanceEventRecord createDropProvenanceEvent(final FlowFileRecord flowFile, final String requestor) { + protected FlowFileRepository getFlowFileRepository() { + return flowFileRepository; + } + + protected ProvenanceEventRepository getProvenanceRepository() { + return provRepository; + } + + protected List createDeleteRepositoryRecords(final List flowFiles) { + final List records = new ArrayList<>(flowFiles.size()); + for (final FlowFileRecord flowFile : flowFiles) { + records.add(createDeleteRepositoryRecord(flowFile)); + } + return records; + } + + protected List createDropProvenanceEvents(final List flowFiles, final String details) { + final List events = new ArrayList<>(flowFiles.size()); + for (final FlowFileRecord flowFile : flowFiles) { + events.add(createDropProvenanceEvent(flowFile, details)); + } + return events; + } + + private ProvenanceEventRecord createDropProvenanceEvent(final FlowFileRecord flowFile, final String details) { final ProvenanceEventBuilder builder = provRepository.eventBuilder(); builder.fromFlowFile(flowFile); builder.setEventType(ProvenanceEventType.DROP); @@ -410,7 +438,7 @@ private ProvenanceEventRecord createDropProvenanceEvent(final FlowFileRecord flo builder.setComponentId(getIdentifier()); builder.setComponentType("Connection"); builder.setAttributes(flowFile.getAttributes(), Collections.emptyMap()); - builder.setDetails("FlowFile Queue emptied by " + requestor); + builder.setDetails(details); builder.setSourceQueueIdentifier(getIdentifier()); final ContentClaim contentClaim = flowFile.getContentClaim(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SelectiveDropResult.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SelectiveDropResult.java new file mode 100644 index 000000000000..3d2e7bf2778b --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SelectiveDropResult.java @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller.queue; + +import org.apache.nifi.controller.repository.FlowFileRecord; + +import java.util.Collections; +import java.util.List; +import java.util.Map; + +/** + * Represents the result of a selective drop operation on a SwappablePriorityQueue. + * Contains the dropped FlowFiles and information about swap file updates. + */ +public class SelectiveDropResult { + + private final List droppedFlowFiles; + private final Map swapLocationUpdates; + + /** + * Creates a new SelectiveDropResult. + * + * @param droppedFlowFiles the FlowFiles that were dropped + * @param swapLocationUpdates a map from old swap location to new swap location. + * If the new location is null, the swap file was deleted entirely. + */ + public SelectiveDropResult(final List droppedFlowFiles, final Map swapLocationUpdates) { + this.droppedFlowFiles = droppedFlowFiles; + this.swapLocationUpdates = swapLocationUpdates; + } + + /** + * @return the FlowFiles that were dropped from the queue + */ + public List getDroppedFlowFiles() { + return Collections.unmodifiableList(droppedFlowFiles); + } + + /** + * @return a map from old swap location to new swap location. If the new location is null, + * all FlowFiles in that swap file matched the predicate and the file should be deleted. + */ + public Map getSwapLocationUpdates() { + return Collections.unmodifiableMap(swapLocationUpdates); + } + + /** + * @return the total number of FlowFiles that were dropped + */ + public int getDroppedCount() { + return droppedFlowFiles.size(); + } + + /** + * @return the total size in bytes of all dropped FlowFiles + */ + public long getDroppedBytes() { + return droppedFlowFiles.stream().mapToLong(FlowFileRecord::getSize).sum(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardFlowFileQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardFlowFileQueue.java index 8e50ab7b7575..8a2a9895d4b6 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardFlowFileQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardFlowFileQueue.java @@ -16,25 +16,34 @@ */ package org.apache.nifi.controller.queue; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.controller.ProcessScheduler; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.FlowFileRepository; import org.apache.nifi.controller.repository.FlowFileSwapManager; +import org.apache.nifi.controller.repository.RepositoryRecord; +import org.apache.nifi.controller.repository.RepositoryRecordType; +import org.apache.nifi.controller.repository.StandardRepositoryRecord; import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.controller.status.FlowFileAvailability; import org.apache.nifi.events.EventReporter; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.processor.FlowFileFilter; +import org.apache.nifi.provenance.ProvenanceEventRecord; import org.apache.nifi.provenance.ProvenanceEventRepository; import org.apache.nifi.util.concurrency.TimedLock; import java.io.IOException; +import java.util.ArrayList; import java.util.Collection; import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.Set; import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReentrantReadWriteLock; +import java.util.function.Predicate; /** * A FlowFileQueue is used to queue FlowFile objects that are awaiting further @@ -203,6 +212,54 @@ protected void dropFlowFiles(final DropFlowFileRequest dropRequest, final String queue.dropFlowFiles(dropRequest, requestor); } + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + lock(); + try { + // Perform the selective drop on the queue, which returns the dropped FlowFiles and swap location updates + final SelectiveDropResult dropResult = queue.dropFlowFiles(predicate); + + if (dropResult.getDroppedCount() == 0) { + return new DropFlowFileSummary(0, 0L); + } + + // Create repository records for the dropped FlowFiles + final List droppedFlowFiles = dropResult.getDroppedFlowFiles(); + final List repositoryRecords = new ArrayList<>(createDeleteRepositoryRecords(droppedFlowFiles)); + + // Create repository records for swap file changes so the FlowFile Repository can track valid swap locations + for (final Map.Entry entry : dropResult.getSwapLocationUpdates().entrySet()) { + final String oldSwapLocation = entry.getKey(); + final String newSwapLocation = entry.getValue(); + + final StandardRepositoryRecord swapRecord = new StandardRepositoryRecord(this); + if (newSwapLocation == null) { + swapRecord.setSwapLocation(oldSwapLocation, RepositoryRecordType.SWAP_FILE_DELETED); + } else { + swapRecord.setSwapFileRenamed(oldSwapLocation, newSwapLocation); + } + repositoryRecords.add(swapRecord); + } + + // Update the FlowFile Repository + getFlowFileRepository().updateRepository(repositoryRecords); + + // Create and register provenance events + final List provenanceEvents = createDropProvenanceEvents(droppedFlowFiles, "Selective drop by predicate"); + getProvenanceRepository().registerEvents(provenanceEvents); + + // Delete old swap files that were replaced + for (final Map.Entry entry : dropResult.getSwapLocationUpdates().entrySet()) { + final String oldSwapLocation = entry.getKey(); + swapManager.deleteSwapFile(oldSwapLocation); + } + + return new DropFlowFileSummary(dropResult.getDroppedCount(), dropResult.getDroppedBytes()); + } finally { + unlock(); + } + } + /** * Lock the queue so that other threads are unable to interact with the queue */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardLocalQueuePartitionDiagnostics.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardLocalQueuePartitionDiagnostics.java index ff31e77e89be..b376f7ab72b4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardLocalQueuePartitionDiagnostics.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/StandardLocalQueuePartitionDiagnostics.java @@ -21,11 +21,16 @@ public class StandardLocalQueuePartitionDiagnostics implements LocalQueuePartiti private final FlowFileQueueSize queueSize; private final boolean anyPenalized; private final boolean allPenalized; + private final QueueSize penalizedQueueSize; + private final QueueSize totalSwapFileQueueSize; - public StandardLocalQueuePartitionDiagnostics(final FlowFileQueueSize queueSize, final boolean anyPenalized, final boolean allPenalized) { + public StandardLocalQueuePartitionDiagnostics(final FlowFileQueueSize queueSize, final boolean anyPenalized, final boolean allPenalized, + final QueueSize penalizedQueueSize, final QueueSize totalSwapFileQueueSize) { this.queueSize = queueSize; this.anyPenalized = anyPenalized; this.allPenalized = allPenalized; + this.penalizedQueueSize = penalizedQueueSize; + this.totalSwapFileQueueSize = totalSwapFileQueueSize; } @Override @@ -57,4 +62,14 @@ public boolean isAnyActiveFlowFilePenalized() { public boolean isAllActiveFlowFilesPenalized() { return allPenalized; } + + @Override + public QueueSize getPenalizedQueueSize() { + return penalizedQueueSize; + } + + @Override + public QueueSize getTotalSwapFileQueueSize() { + return totalSwapFileQueueSize; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java index 694679770048..827a2a29fef1 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java @@ -43,6 +43,7 @@ import java.util.Collections; import java.util.HashMap; import java.util.Iterator; +import java.util.LinkedHashMap; import java.util.LinkedHashSet; import java.util.List; import java.util.Map; @@ -52,13 +53,15 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicReference; import java.util.concurrent.locks.ReentrantReadWriteLock; +import java.util.function.Predicate; public class SwappablePriorityQueue { private static final Logger logger = LoggerFactory.getLogger(SwappablePriorityQueue.class); - private static final int SWAP_RECORD_POLL_SIZE = 10_000; + private static final int MAX_SWAP_RECORD_POLL_SIZE = 10_000; private static final int MAX_EXPIRED_RECORDS_PER_ITERATION = 10_000; private final int swapThreshold; + private final int swapRecordPollSize; private final FlowFileSwapManager swapManager; private final EventReporter eventReporter; private final FlowFileQueue flowFileQueue; @@ -74,7 +77,7 @@ public class SwappablePriorityQueue { // We keep an "active queue" and a "swap queue" that both are able to hold records in heap. When // FlowFiles are added to this FlowFileQueue, we first check if we are in "swap mode" and if so // we add to the 'swap queue' instead of the 'active queue'. The code would be much simpler if we - // eliminated the 'swap queue' and instead just used the active queue and swapped out the 10,000 + // eliminated the 'swap queue' and instead just used the active queue and swapped out the // lowest priority FlowFiles from that. However, doing that would cause problems with the ordering // of FlowFiles. If we swap out some FlowFiles, and then allow a new FlowFile to be written to the // active queue, then we would end up processing the newer FlowFile before the swapped FlowFile. By @@ -95,6 +98,7 @@ public SwappablePriorityQueue(final FlowFileSwapManager swapManager, final int s final DropFlowFileAction dropAction, final String swapPartitionName) { this.swapManager = swapManager; this.swapThreshold = swapThreshold; + this.swapRecordPollSize = Math.min(MAX_SWAP_RECORD_POLL_SIZE, swapThreshold); this.activeQueue = new PriorityQueue<>(20, new QueuePrioritizer(Collections.emptyList())); this.swapQueue = new ArrayList<>(); @@ -132,10 +136,34 @@ public void setPriorities(final List newPriorities) { public LocalQueuePartitionDiagnostics getQueueDiagnostics() { readLock.lock(); try { - final boolean anyPenalized = !activeQueue.isEmpty() && activeQueue.peek().isPenalized(); - final boolean allPenalized = anyPenalized && activeQueue.stream().anyMatch(FlowFileRecord::isPenalized); + int penalizedCount = 0; + long penalizedBytes = 0; + for (final FlowFileRecord flowFile : activeQueue) { + if (flowFile.isPenalized()) { + penalizedCount++; + penalizedBytes += flowFile.getSize(); + } + } + + final boolean anyPenalized = penalizedCount > 0; + final boolean allPenalized = anyPenalized && penalizedCount == activeQueue.size(); + final QueueSize penalizedQueueSize = new QueueSize(penalizedCount, penalizedBytes); + + int totalSwapFlowFiles = 0; + long totalSwapBytes = 0; + for (final String swapLocation : swapLocations) { + try { + final SwapSummary summary = swapManager.getSwapSummary(swapLocation); + final QueueSize queueSize = summary.getQueueSize(); + totalSwapFlowFiles += queueSize.getObjectCount(); + totalSwapBytes += queueSize.getByteCount(); + } catch (final IOException e) { + logger.warn("Unable to read swap file summary for {}: {}", swapLocation, e.getMessage()); + } + } + final QueueSize totalSwapFileQueueSize = new QueueSize(totalSwapFlowFiles, totalSwapBytes); - return new StandardLocalQueuePartitionDiagnostics(getFlowFileQueueSize(), anyPenalized, allPenalized); + return new StandardLocalQueuePartitionDiagnostics(getFlowFileQueueSize(), anyPenalized, allPenalized, penalizedQueueSize, totalSwapFileQueueSize); } finally { readLock.unlock("getQueueDiagnostics"); } @@ -158,16 +186,16 @@ public boolean isUnacknowledgedFlowFile() { * This method MUST be called with the write lock held */ private void writeSwapFilesIfNecessary() { - if (swapQueue.size() < SWAP_RECORD_POLL_SIZE) { + if (swapQueue.size() < swapRecordPollSize) { return; } migrateSwapToActive(); - if (swapQueue.size() < SWAP_RECORD_POLL_SIZE) { + if (swapQueue.size() < swapRecordPollSize) { return; } - final int numSwapFiles = swapQueue.size() / SWAP_RECORD_POLL_SIZE; + final int numSwapFiles = swapQueue.size() / swapRecordPollSize; int originalSwapQueueCount = swapQueue.size(); long originalSwapQueueBytes = 0L; @@ -190,9 +218,9 @@ private void writeSwapFilesIfNecessary() { long totalSwapQueueDatesThisIteration = 0L; long minQueueDateThisIteration = Long.MAX_VALUE; - // Create a new swap file for the next SWAP_RECORD_POLL_SIZE records - final List toSwap = new ArrayList<>(SWAP_RECORD_POLL_SIZE); - for (int j = 0; j < SWAP_RECORD_POLL_SIZE; j++) { + // Create a new swap file for the next swapRecordPollSize records + final List toSwap = new ArrayList<>(swapRecordPollSize); + for (int j = 0; j < swapRecordPollSize; j++) { final FlowFileRecord flowFile = tempQueue.poll(); toSwap.add(flowFile); bytesSwappedThisIteration += flowFile.getSize(); @@ -894,6 +922,210 @@ public void dropFlowFiles(final DropFlowFileRequest dropRequest, final String re } } + /** + * Performs a selective drop of FlowFiles that match the given predicate. This method: + * 1. Filters the active queue and swap queue, collecting matched FlowFiles + * 2. For each swap file, peeks to deserialize FlowFiles, filters them, and writes remaining + * FlowFiles to a new swap file (if any remain) + * 3. Returns a SelectiveDropResult containing all dropped FlowFiles and swap location updates + * + * Note: This method does NOT update the FlowFile Repository or Provenance Repository. + * The caller is responsible for updating the repositories after this method returns. + * + * @param predicate the predicate to determine which FlowFiles to drop + * @return a SelectiveDropResult containing dropped FlowFiles and swap location updates + * @throws IOException if an error occurs while reading or writing swap files + */ + public SelectiveDropResult dropFlowFiles(final Predicate predicate) throws IOException { + final List droppedFlowFiles = new ArrayList<>(); + final Map swapLocationUpdates = new LinkedHashMap<>(); + + writeLock.lock(); + try { + // Process swap files FIRST - this is done first because it can throw IOException. + // By processing swap files first, we avoid modifying in-memory state if swap file operations fail. + // We use a two-phase approach: first create all new swap files, then if all succeed, update state. + + // Phase 1: Process all swap files and create new ones as needed + // Track what we need to do for each swap file + final List swapFileResults = new ArrayList<>(); + final List newlyCreatedSwapFiles = new ArrayList<>(); + + try { + for (final String swapLocation : swapLocations) { + final SwapFileDropResult result = processSwapFileForDrop(swapLocation, predicate); + if (result != null) { + swapFileResults.add(result); + if (result.newSwapLocation() != null) { + newlyCreatedSwapFiles.add(result.newSwapLocation()); + } + } + } + } catch (final IOException ioe) { + // Failed to process swap files - delete all newly created swap files and throw exception + logger.error("Failed to process swap files during selective drop; rolling back all changes", ioe); + for (final String newSwapFile : newlyCreatedSwapFiles) { + try { + swapManager.deleteSwapFile(newSwapFile); + logger.debug("Deleted newly created swap file {} during rollback", newSwapFile); + } catch (final Exception deleteException) { + logger.warn("Failed to delete newly created swap file {} during rollback", newSwapFile, deleteException); + } + } + throw ioe; + } + + // Phase 2: All swap file operations succeeded - now update state + for (final SwapFileDropResult result : swapFileResults) { + final String oldSwapLocation = result.oldSwapLocation(); + final String newSwapLocation = result.newSwapLocation(); + + swapLocationUpdates.put(oldSwapLocation, newSwapLocation); + swapLocations.remove(oldSwapLocation); + + if (newSwapLocation != null) { + // Some FlowFiles remain in new swap file + swapLocations.add(newSwapLocation); + incrementSwapQueueSize(-result.droppedFlowFiles().size(), -result.droppedBytes(), 0); + + // Update metrics for the new swap location + minQueueDateInSwapLocation.put(newSwapLocation, result.remainingMinQueueDate()); + totalQueueDateInSwapLocation.put(newSwapLocation, result.remainingTotalQueueDate()); + + logger.debug("Selective drop removed {} FlowFiles from swap file {}, wrote remaining to {}", + result.droppedFlowFiles().size(), oldSwapLocation, newSwapLocation); + } else { + // All FlowFiles were dropped - swap file count decreases + incrementSwapQueueSize(-result.droppedFlowFiles().size(), -result.droppedBytes(), -1); + logger.debug("Selective drop removed all {} FlowFiles from swap file {}", result.droppedFlowFiles().size(), oldSwapLocation); + } + + // Remove metrics for the old swap location + minQueueDateInSwapLocation.remove(oldSwapLocation); + totalQueueDateInSwapLocation.remove(oldSwapLocation); + + droppedFlowFiles.addAll(result.droppedFlowFiles()); + } + + // Filter the active queue + final Queue newActiveQueue = new PriorityQueue<>(Math.max(20, activeQueue.size()), new QueuePrioritizer(getPriorities())); + int droppedFromActiveCount = 0; + long droppedFromActiveBytes = 0L; + + for (final FlowFileRecord flowFile : activeQueue) { + if (predicate.test(flowFile)) { + droppedFlowFiles.add(flowFile); + droppedFromActiveCount++; + droppedFromActiveBytes += flowFile.getSize(); + } else { + newActiveQueue.add(flowFile); + } + } + activeQueue = newActiveQueue; + if (droppedFromActiveCount > 0) { + incrementActiveQueueSize(-droppedFromActiveCount, -droppedFromActiveBytes); + logger.debug("Selective drop removed {} FlowFiles ({} bytes) from active queue", droppedFromActiveCount, droppedFromActiveBytes); + } + + // Filter the swap queue + final List newSwapQueue = new ArrayList<>(); + int droppedFromSwapQueueCount = 0; + long droppedFromSwapQueueBytes = 0L; + + for (final FlowFileRecord flowFile : swapQueue) { + if (predicate.test(flowFile)) { + droppedFlowFiles.add(flowFile); + droppedFromSwapQueueCount++; + droppedFromSwapQueueBytes += flowFile.getSize(); + } else { + newSwapQueue.add(flowFile); + } + } + swapQueue = newSwapQueue; + if (droppedFromSwapQueueCount > 0) { + incrementSwapQueueSize(-droppedFromSwapQueueCount, -droppedFromSwapQueueBytes, 0); + logger.debug("Selective drop removed {} FlowFiles ({} bytes) from swap queue", droppedFromSwapQueueCount, droppedFromSwapQueueBytes); + } + + // Update swap mode if we've drained all queues + if (swapQueue.isEmpty() && swapLocations.isEmpty()) { + swapMode = false; + } + + updateTopPenaltyExpiration(); + + logger.info("Selective drop completed for queue {}: dropped {} FlowFiles ({} bytes)", + getQueueIdentifier(), droppedFlowFiles.size(), droppedFlowFiles.stream().mapToLong(FlowFileRecord::getSize).sum()); + + return new SelectiveDropResult(droppedFlowFiles, swapLocationUpdates); + } finally { + writeLock.unlock("Selective Drop FlowFiles"); + } + } + + /** + * Processes a single swap file for selective drop, filtering FlowFiles based on the predicate. + * If any FlowFiles match the predicate, a new swap file is created with the remaining FlowFiles + * (or the swap file is marked for deletion if all FlowFiles match). + * + * @param swapLocation the location of the swap file to process + * @param predicate the predicate to determine which FlowFiles to drop + * @return a SwapFileDropResult if any FlowFiles were dropped, or null if no changes are needed + * @throws IOException if an error occurs reading or writing swap files + */ + private SwapFileDropResult processSwapFileForDrop(final String swapLocation, final Predicate predicate) throws IOException { + final SwapContents swapContents; + try { + swapContents = swapManager.peek(swapLocation, flowFileQueue); + } catch (final IncompleteSwapFileException isfe) { + logger.warn("Failed to read swap file {} due to incomplete file; some FlowFiles may not be filtered", swapLocation); + if (eventReporter != null) { + eventReporter.reportEvent(Severity.WARNING, "Selective Drop", "Failed to read swap file " + swapLocation + + " because the file was incomplete. Some FlowFiles may not be filtered."); + } + return null; + } + + final List swappedFlowFiles = swapContents.getFlowFiles(); + final List remainingFlowFiles = new ArrayList<>(); + final List droppedFromThisSwapFile = new ArrayList<>(); + long droppedFromSwapFileBytes = 0L; + long remainingMinQueueDate = Long.MAX_VALUE; + long remainingTotalQueueDate = 0L; + + for (final FlowFileRecord flowFile : swappedFlowFiles) { + if (predicate.test(flowFile)) { + droppedFromThisSwapFile.add(flowFile); + droppedFromSwapFileBytes += flowFile.getSize(); + } else { + remainingFlowFiles.add(flowFile); + remainingMinQueueDate = Math.min(remainingMinQueueDate, flowFile.getLastQueueDate()); + remainingTotalQueueDate += flowFile.getLastQueueDate(); + } + } + + if (droppedFromThisSwapFile.isEmpty()) { + return null; + } + + if (remainingFlowFiles.isEmpty()) { + // All FlowFiles matched - swap file will be deleted + return new SwapFileDropResult(swapLocation, null, droppedFromThisSwapFile, droppedFromSwapFileBytes, 0L, 0L); + } + + // Some FlowFiles remain - write to new swap file + final String newSwapLocation = swapManager.swapOut(remainingFlowFiles, flowFileQueue, swapPartitionName); + return new SwapFileDropResult(swapLocation, newSwapLocation, droppedFromThisSwapFile, + droppedFromSwapFileBytes, remainingMinQueueDate, remainingTotalQueueDate); + } + + /** + * Holds the result of processing a single swap file during selective drop. + */ + private record SwapFileDropResult(String oldSwapLocation, String newSwapLocation, List droppedFlowFiles, + long droppedBytes, long remainingMinQueueDate, long remainingTotalQueueDate) { + } + public SwapSummary recoverSwappedFlowFiles() { int swapFlowFileCount = 0; long swapByteCount = 0L; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java index 51b68806e377..39c1c0c9b656 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java @@ -22,6 +22,7 @@ import org.apache.nifi.cluster.coordination.node.NodeConnectionState; import org.apache.nifi.cluster.coordination.node.NodeConnectionStatus; import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.controller.ProcessScheduler; import org.apache.nifi.controller.queue.AbstractFlowFileQueue; import org.apache.nifi.controller.queue.DropFlowFileRequest; @@ -35,6 +36,7 @@ import org.apache.nifi.controller.queue.QueueDiagnostics; import org.apache.nifi.controller.queue.QueueSize; import org.apache.nifi.controller.queue.RemoteQueuePartitionDiagnostics; +import org.apache.nifi.controller.queue.SelectiveDropResult; import org.apache.nifi.controller.queue.StandardQueueDiagnostics; import org.apache.nifi.controller.queue.SwappablePriorityQueue; import org.apache.nifi.controller.queue.clustered.client.async.AsyncLoadBalanceClientRegistry; @@ -55,6 +57,7 @@ import org.apache.nifi.controller.repository.FlowFileRepository; import org.apache.nifi.controller.repository.FlowFileSwapManager; import org.apache.nifi.controller.repository.RepositoryRecord; +import org.apache.nifi.controller.repository.RepositoryRecordType; import org.apache.nifi.controller.repository.StandardRepositoryRecord; import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.controller.repository.claim.ContentClaim; @@ -62,6 +65,7 @@ import org.apache.nifi.controller.status.FlowFileAvailability; import org.apache.nifi.controller.swap.StandardSwapSummary; import org.apache.nifi.events.EventReporter; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.processor.FlowFileFilter; import org.apache.nifi.provenance.ProvenanceEventBuilder; @@ -91,6 +95,7 @@ import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.function.Function; +import java.util.function.Predicate; import java.util.stream.Collectors; public class SocketLoadBalancedFlowFileQueue extends AbstractFlowFileQueue implements LoadBalancedFlowFileQueue { @@ -1098,6 +1103,67 @@ protected void dropFlowFiles(final DropFlowFileRequest dropRequest, final String } } + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + lock(); + try { + final List allDroppedFlowFiles = new ArrayList<>(); + final Map allSwapLocationUpdates = new HashMap<>(); + + for (final QueuePartition partition : queuePartitions) { + final SelectiveDropResult partitionResult = partition.dropFlowFiles(predicate); + allDroppedFlowFiles.addAll(partitionResult.getDroppedFlowFiles()); + allSwapLocationUpdates.putAll(partitionResult.getSwapLocationUpdates()); + adjustSize(-partitionResult.getDroppedCount(), -partitionResult.getDroppedBytes()); + } + + // Also drop from the rebalancing partition + final SelectiveDropResult rebalanceResult = rebalancingPartition.dropFlowFiles(predicate); + allDroppedFlowFiles.addAll(rebalanceResult.getDroppedFlowFiles()); + allSwapLocationUpdates.putAll(rebalanceResult.getSwapLocationUpdates()); + adjustSize(-rebalanceResult.getDroppedCount(), -rebalanceResult.getDroppedBytes()); + + if (allDroppedFlowFiles.isEmpty()) { + return new DropFlowFileSummary(0, 0L); + } + + // Create repository records for the dropped FlowFiles + final List repositoryRecords = new ArrayList<>(createDeleteRepositoryRecords(allDroppedFlowFiles)); + + // Create repository records for swap file changes so the FlowFile Repository can track valid swap locations + for (final Map.Entry entry : allSwapLocationUpdates.entrySet()) { + final String oldSwapLocation = entry.getKey(); + final String newSwapLocation = entry.getValue(); + + final StandardRepositoryRecord swapRecord = new StandardRepositoryRecord(this); + if (newSwapLocation == null) { + swapRecord.setSwapLocation(oldSwapLocation, RepositoryRecordType.SWAP_FILE_DELETED); + } else { + swapRecord.setSwapFileRenamed(oldSwapLocation, newSwapLocation); + } + repositoryRecords.add(swapRecord); + } + + // Update the FlowFile Repository + flowFileRepo.updateRepository(repositoryRecords); + + // Create and register provenance events + final List provenanceEvents = createDropProvenanceEvents(allDroppedFlowFiles, "Selective drop by predicate"); + provRepo.registerEvents(provenanceEvents); + + // Delete old swap files that were replaced + for (final String oldSwapLocation : allSwapLocationUpdates.keySet()) { + swapManager.deleteSwapFile(oldSwapLocation); + } + + final int totalDroppedCount = allDroppedFlowFiles.size(); + final long totalDroppedBytes = allDroppedFlowFiles.stream().mapToLong(FlowFileRecord::getSize).sum(); + return new DropFlowFileSummary(totalDroppedCount, totalDroppedBytes); + } finally { + unlock(); + } + } + @Override public void lock() { partitionReadLock.lock(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/QueuePartition.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/QueuePartition.java index 60259d111b4b..2bf59de2808f 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/QueuePartition.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/QueuePartition.java @@ -21,13 +21,17 @@ import org.apache.nifi.controller.queue.DropFlowFileRequest; import org.apache.nifi.controller.queue.FlowFileQueueContents; import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.queue.SelectiveDropResult; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.SwapSummary; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; +import java.io.IOException; import java.util.Collection; import java.util.List; import java.util.Optional; +import java.util.function.Predicate; /** * Represents a portion of a FlowFile Queue such that a FlowFile Queue can be broken into @@ -69,6 +73,15 @@ public interface QueuePartition { */ void dropFlowFiles(DropFlowFileRequest dropRequest, String requestor); + /** + * Drops all FlowFiles in this partition that match the given predicate. + * + * @param predicate the predicate used to determine which FlowFiles should be dropped + * @return a SelectiveDropResult containing the dropped FlowFiles and swap location updates + * @throws IOException if an error occurs while reading or writing swap files + */ + SelectiveDropResult dropFlowFiles(Predicate predicate) throws IOException; + /** * Updates the prioritizers to use when queueing data * @param newPriorities the new priorities diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RemoteQueuePartition.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RemoteQueuePartition.java index b73f2c279654..277d50b22e08 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RemoteQueuePartition.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RemoteQueuePartition.java @@ -24,6 +24,7 @@ import org.apache.nifi.controller.queue.PollStrategy; import org.apache.nifi.controller.queue.QueueSize; import org.apache.nifi.controller.queue.RemoteQueuePartitionDiagnostics; +import org.apache.nifi.controller.queue.SelectiveDropResult; import org.apache.nifi.controller.queue.StandardRemoteQueuePartitionDiagnostics; import org.apache.nifi.controller.queue.SwappablePriorityQueue; import org.apache.nifi.controller.queue.clustered.TransferFailureDestination; @@ -39,6 +40,7 @@ import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.controller.repository.claim.ContentClaim; import org.apache.nifi.controller.repository.claim.ResourceClaim; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.flowfile.attributes.CoreAttributes; import org.apache.nifi.provenance.ProvenanceEventBuilder; @@ -49,6 +51,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.IOException; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; @@ -58,6 +61,7 @@ import java.util.Set; import java.util.concurrent.TimeUnit; import java.util.function.BooleanSupplier; +import java.util.function.Predicate; import java.util.stream.Collectors; /** @@ -135,6 +139,11 @@ public void dropFlowFiles(final DropFlowFileRequest dropRequest, final String re priorityQueue.dropFlowFiles(dropRequest, requestor); } + @Override + public SelectiveDropResult dropFlowFiles(final Predicate predicate) throws IOException { + return priorityQueue.dropFlowFiles(predicate); + } + @Override public SwapSummary recoverSwappedFlowFiles() { return priorityQueue.recoverSwappedFlowFiles(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/StandardRebalancingPartition.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/StandardRebalancingPartition.java index 439cd8374a05..9fa3dced3eb5 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/StandardRebalancingPartition.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/StandardRebalancingPartition.java @@ -25,20 +25,24 @@ import org.apache.nifi.controller.queue.LoadBalancedFlowFileQueue; import org.apache.nifi.controller.queue.PollStrategy; import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.queue.SelectiveDropResult; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.FlowFileSwapManager; import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.events.EventReporter; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.IOException; import java.util.ArrayList; import java.util.Collection; import java.util.HashSet; import java.util.List; import java.util.Optional; import java.util.Set; +import java.util.function.Predicate; public class StandardRebalancingPartition implements RebalancingPartition { private static final Logger logger = LoggerFactory.getLogger(StandardRebalancingPartition.class); @@ -106,6 +110,11 @@ public void dropFlowFiles(DropFlowFileRequest dropRequest, String requestor) { queue.dropFlowFiles(dropRequest, requestor); } + @Override + public SelectiveDropResult dropFlowFiles(final Predicate predicate) throws IOException { + return queue.dropFlowFiles(predicate); + } + @Override public void setPriorities(final List newPriorities) { queue.setPriorities(newPriorities); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/SwappablePriorityQueueLocalPartition.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/SwappablePriorityQueueLocalPartition.java index 4674cce74e07..5d84760b4a51 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/SwappablePriorityQueueLocalPartition.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/SwappablePriorityQueueLocalPartition.java @@ -25,12 +25,14 @@ import org.apache.nifi.controller.queue.LocalQueuePartitionDiagnostics; import org.apache.nifi.controller.queue.PollStrategy; import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.queue.SelectiveDropResult; import org.apache.nifi.controller.queue.SwappablePriorityQueue; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.FlowFileSwapManager; import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.controller.status.FlowFileAvailability; import org.apache.nifi.events.EventReporter; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.processor.FlowFileFilter; @@ -40,6 +42,7 @@ import java.util.Optional; import java.util.Set; import java.util.concurrent.TimeUnit; +import java.util.function.Predicate; /** * A Local Queue Partition that whose implementation is based on the use of a {@link SwappablePriorityQueue}. @@ -157,6 +160,11 @@ public void dropFlowFiles(final DropFlowFileRequest dropRequest, final String re priorityQueue.dropFlowFiles(dropRequest, requestor); } + @Override + public SelectiveDropResult dropFlowFiles(final Predicate predicate) throws IOException { + return priorityQueue.dropFlowFiles(predicate); + } + @Override public SwapSummary recoverSwappedFlowFiles() { return priorityQueue.recoverSwappedFlowFiles(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/reporting/StandardReportingTaskNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/reporting/StandardReportingTaskNode.java index dfaf6347309f..055a1d246eb0 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/reporting/StandardReportingTaskNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/reporting/StandardReportingTaskNode.java @@ -22,6 +22,7 @@ import org.apache.nifi.authorization.resource.Authorizable; import org.apache.nifi.authorization.resource.ResourceFactory; import org.apache.nifi.authorization.resource.ResourceType; +import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.validation.ValidationTrigger; import org.apache.nifi.controller.FlowController; @@ -91,7 +92,7 @@ public ReportingContext getReportingContext() { } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/LiveSerializedRepositoryRecord.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/LiveSerializedRepositoryRecord.java index 01df125ac2bc..f5176e9ad846 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/LiveSerializedRepositoryRecord.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/LiveSerializedRepositoryRecord.java @@ -54,6 +54,11 @@ public String getSwapLocation() { return record.getSwapLocation(); } + @Override + public String getOriginalSwapLocation() { + return record.getOriginalSwapLocation(); + } + @Override public FlowFileRecord getFlowFileRecord() { return record.getCurrent(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java index a04b7527917d..aa880e8ba37a 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java @@ -418,7 +418,9 @@ public boolean isValidSwapLocationSuffix(final String swapLocationSuffix) { private void updateRepository(final Collection records, final boolean sync) throws IOException { for (final RepositoryRecord record : records) { if (record.getType() != RepositoryRecordType.DELETE && record.getType() != RepositoryRecordType.CONTENTMISSING - && record.getType() != RepositoryRecordType.CLEANUP_TRANSIENT_CLAIMS && record.getDestination() == null) { + && record.getType() != RepositoryRecordType.CLEANUP_TRANSIENT_CLAIMS + && record.getType() != RepositoryRecordType.SWAP_FILE_DELETED && record.getType() != RepositoryRecordType.SWAP_FILE_RENAMED + && record.getDestination() == null) { throw new IllegalArgumentException("Record " + record + " has no destination and Type is " + record.getType()); } } @@ -481,6 +483,17 @@ protected void updateContentClaims(Collection repositoryRecord final String swapLocation = record.getSwapLocation(); swapLocationsRemoved.add(swapLocation); swapLocationsAdded.remove(swapLocation); + } else if (record.getType() == RepositoryRecordType.SWAP_FILE_DELETED) { + final String swapLocation = record.getSwapLocation(); + swapLocationsRemoved.add(swapLocation); + swapLocationsAdded.remove(swapLocation); + } else if (record.getType() == RepositoryRecordType.SWAP_FILE_RENAMED) { + final String originalSwapLocation = record.getOriginalSwapLocation(); + final String newSwapLocation = record.getSwapLocation(); + swapLocationsRemoved.add(originalSwapLocation); + swapLocationsAdded.remove(originalSwapLocation); + swapLocationsAdded.add(newSwapLocation); + swapLocationsRemoved.remove(newSwapLocation); } } @@ -635,7 +648,7 @@ public void swapFlowFilesIn(final String swapLocation, final List recoveredSwapLocations = wal.getRecoveredSwapLocations(); synchronized (this.swapLocationSuffixes) { recoveredSwapLocations.forEach(loc -> this.swapLocationSuffixes.add(normalizeSwapLocation(loc))); + + for (final SerializedRepositoryRecord record : recordList) { + final RepositoryRecordType recordType = record.getType(); + if (recordType == RepositoryRecordType.SWAP_FILE_DELETED) { + this.swapLocationSuffixes.remove(normalizeSwapLocation(record.getSwapLocation())); + } else if (recordType == RepositoryRecordType.SWAP_FILE_RENAMED) { + this.swapLocationSuffixes.remove(normalizeSwapLocation(record.getOriginalSwapLocation())); + this.swapLocationSuffixes.add(normalizeSwapLocation(record.getSwapLocation())); + } + } + logger.debug("Recovered {} Swap Files: {}", swapLocationSuffixes.size(), swapLocationSuffixes); } @@ -727,6 +751,11 @@ public long loadFlowFiles(final QueueProvider queueProvider) throws IOException int numFlowFilesMissingQueue = 0; long maxId = 0; for (final SerializedRepositoryRecord record : recordList) { + final RepositoryRecordType recordType = record.getType(); + if (recordType == RepositoryRecordType.SWAP_FILE_DELETED || recordType == RepositoryRecordType.SWAP_FILE_RENAMED) { + continue; + } + final long recordId = serdeFactory.getRecordIdentifier(record); if (recordId > maxId) { maxId = recordId; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/scheduling/StandardProcessScheduler.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/scheduling/StandardProcessScheduler.java index be1982054f4d..6c5645b28aa4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/scheduling/StandardProcessScheduler.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/scheduling/StandardProcessScheduler.java @@ -22,6 +22,7 @@ import org.apache.nifi.annotation.lifecycle.OnUnscheduled; import org.apache.nifi.annotation.notification.PrimaryNodeState; import org.apache.nifi.authorization.resource.ComponentAuthorizable; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.components.state.StateManager; import org.apache.nifi.components.state.StateManagerProvider; import org.apache.nifi.components.validation.ValidationStatus; @@ -31,8 +32,10 @@ import org.apache.nifi.controller.AbstractPort; import org.apache.nifi.controller.ConfigurationContext; import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.NodeTypeProvider; import org.apache.nifi.controller.ProcessScheduler; import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.ReloadComponent; import org.apache.nifi.controller.ReportingTaskNode; import org.apache.nifi.controller.ScheduledState; import org.apache.nifi.controller.SchedulingAgentCallback; @@ -48,6 +51,7 @@ import org.apache.nifi.lifecycle.ProcessorStopLifecycleMethods; import org.apache.nifi.logging.ComponentLog; import org.apache.nifi.logging.StandardLoggingContext; +import org.apache.nifi.nar.ExtensionManager; import org.apache.nifi.nar.NarCloseable; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.Processor; @@ -88,13 +92,16 @@ public final class StandardProcessScheduler implements ProcessScheduler { private static final Logger LOG = LoggerFactory.getLogger(StandardProcessScheduler.class); - private final FlowController flowController; private final long administrativeYieldMillis; private final String administrativeYieldDuration; private final StateManagerProvider stateManagerProvider; private final long processorStartTimeoutMillis; private final LifecycleStateManager lifecycleStateManager; private final AtomicLong frameworkTaskThreadIndex = new AtomicLong(1L); + private final ExtensionManager extensionManager; + private final NodeTypeProvider nodeTypeProvider; + private final Supplier controllerServiceProviderFactory; + private final ReloadComponent reloadComponent; private final ConcurrentMap strategyAgentMap = new ConcurrentHashMap<>(); @@ -104,12 +111,24 @@ public final class StandardProcessScheduler implements ProcessScheduler { private final ScheduledExecutorService componentMonitoringThreadPool = new FlowEngine(2, "Monitor Processor Lifecycle", true); public StandardProcessScheduler(final FlowEngine componentLifecycleThreadPool, final FlowController flowController, + final StateManagerProvider stateManagerProvider, final NiFiProperties nifiProperties, final LifecycleStateManager lifecycleStateManager) { + + this(componentLifecycleThreadPool, flowController.getExtensionManager(), flowController, flowController::getControllerServiceProvider, + flowController.getReloadComponent(), stateManagerProvider, nifiProperties, lifecycleStateManager); + } + + public StandardProcessScheduler(final FlowEngine componentLifecycleThreadPool, final ExtensionManager extensionManager, final NodeTypeProvider nodeTypeProvider, + final Supplier controllerServiceProviderFactory, final ReloadComponent reloadComponent, final StateManagerProvider stateManagerProvider, final NiFiProperties nifiProperties, final LifecycleStateManager lifecycleStateManager) { + this.componentLifeCycleThreadPool = componentLifecycleThreadPool; - this.flowController = flowController; this.stateManagerProvider = stateManagerProvider; this.lifecycleStateManager = lifecycleStateManager; + this.extensionManager = extensionManager; + this.nodeTypeProvider = nodeTypeProvider; + this.controllerServiceProviderFactory = controllerServiceProviderFactory; + this.reloadComponent = reloadComponent; administrativeYieldDuration = nifiProperties.getAdministrativeYieldDuration(); administrativeYieldMillis = FormatUtils.getTimeDuration(administrativeYieldDuration, TimeUnit.MILLISECONDS); @@ -119,7 +138,7 @@ public StandardProcessScheduler(final FlowEngine componentLifecycleThreadPool, f } public ControllerServiceProvider getControllerServiceProvider() { - return flowController.getControllerServiceProvider(); + return controllerServiceProviderFactory.get(); } private StateManager getStateManager(final String componentId) { @@ -239,7 +258,7 @@ public void shutdown() { @Override public void shutdownReportingTask(final ReportingTaskNode reportingTask) { final ConfigurationContext configContext = reportingTask.getConfigurationContext(); - try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(flowController.getExtensionManager(), reportingTask.getReportingTask().getClass(), reportingTask.getIdentifier())) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, reportingTask.getReportingTask().getClass(), reportingTask.getIdentifier())) { ReflectionUtils.quietlyInvokeMethodsWithAnnotation(OnShutdown.class, reportingTask.getReportingTask(), configContext); } } @@ -247,7 +266,7 @@ public void shutdownReportingTask(final ReportingTaskNode reportingTask) { @Override public void shutdownControllerService(final ControllerServiceNode serviceNode, final ControllerServiceProvider controllerServiceProvider) { final Class serviceImplClass = serviceNode.getControllerServiceImplementation().getClass(); - try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(flowController.getExtensionManager(), serviceImplClass, serviceNode.getIdentifier())) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, serviceImplClass, serviceNode.getIdentifier())) { final ConfigurationContext configContext = new StandardConfigurationContext(serviceNode, controllerServiceProvider, null); ReflectionUtils.quietlyInvokeMethodsWithAnnotation(OnShutdown.class, serviceNode.getControllerServiceImplementation(), configContext); } @@ -295,7 +314,7 @@ public void run() { return; } - try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(flowController.getExtensionManager(), reportingTask.getClass(), reportingTask.getIdentifier())) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, reportingTask.getClass(), reportingTask.getIdentifier())) { ReflectionUtils.invokeMethodsWithAnnotation(OnScheduled.class, reportingTask, taskNode.getConfigurationContext()); } @@ -310,7 +329,7 @@ public void run() { + "ReportingTask and will attempt to schedule it again after {}", reportingTask, e.toString(), administrativeYieldDuration, e); - try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(flowController.getExtensionManager(), reportingTask.getClass(), reportingTask.getIdentifier())) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, reportingTask.getClass(), reportingTask.getIdentifier())) { ReflectionUtils.quietlyInvokeMethodsWithAnnotation(OnUnscheduled.class, reportingTask, taskNode.getConfigurationContext()); ReflectionUtils.quietlyInvokeMethodsWithAnnotation(OnStopped.class, reportingTask, taskNode.getConfigurationContext()); } @@ -349,7 +368,7 @@ public Future unschedule(final ReportingTaskNode taskNode) { synchronized (lifecycleState) { lifecycleState.setScheduled(false); - try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(flowController.getExtensionManager(), reportingTask.getClass(), reportingTask.getIdentifier())) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, reportingTask.getClass(), reportingTask.getIdentifier())) { ReflectionUtils.invokeMethodsWithAnnotation(OnUnscheduled.class, reportingTask, configurationContext); } catch (final Exception e) { final Throwable cause = e instanceof InvocationTargetException ? e.getCause() : e; @@ -390,7 +409,7 @@ public synchronized CompletableFuture startProcessor(final ProcessorNode p final LifecycleState lifecycleState = getLifecycleState(requireNonNull(procNode), true, false); final Supplier processContextFactory = () -> new StandardProcessContext(procNode, getControllerServiceProvider(), - getStateManager(procNode), lifecycleState::isTerminated, flowController); + getStateManager(procNode), lifecycleState::isTerminated, nodeTypeProvider); final boolean scheduleActions = procNode.getProcessGroup().resolveExecutionEngine() != ExecutionEngine.STATELESS; @@ -421,12 +440,21 @@ public void onTaskComplete() { }; LOG.info("Starting {}", procNode); + + procNode.reloadAdditionalResourcesIfNecessary(); + procNode.start(componentMonitoringThreadPool, administrativeYieldMillis, processorStartTimeoutMillis, processContextFactory, callback, failIfStopping, scheduleActions); return future; } @Override public synchronized CompletableFuture startStatelessGroup(final StatelessGroupNode groupNode) { + final ScheduledState initialState = groupNode.getDesiredState(); + if (initialState != ScheduledState.RUNNING) { + LOG.warn("{} scheduled to run but its desired state is {}, not RUNNING; will not start the group", groupNode, initialState); + return CompletableFuture.failedFuture(new IllegalStateException("Attempted to start %s but its desired state is %s, not RUNNING".formatted(groupNode, initialState))); + } + final LifecycleState lifecycleState = getLifecycleState(requireNonNull(groupNode), true, true); lifecycleState.setScheduled(true); @@ -495,7 +523,7 @@ public void trigger() { } } - LOG.info("{} is no scheduled to run. Disabling {} Controller Services", groupNode.getProcessGroup(), allServices.size()); + LOG.info("{} is no longer scheduled to run. Disabling {} Controller Services", groupNode.getProcessGroup(), allServices.size()); // Cancel all service start futures, interrupting them if they are waiting for (final Future serviceStartFuture : serviceStartFutures) { @@ -523,7 +551,7 @@ public Future runProcessorOnce(ProcessorNode procNode, final Callable processContextFactory = () -> new StandardProcessContext(procNode, getControllerServiceProvider(), - getStateManager(procNode), lifecycleState::isTerminated, flowController); + getStateManager(procNode), lifecycleState::isTerminated, nodeTypeProvider); final CompletableFuture future = new CompletableFuture<>(); final SchedulingAgentCallback callback = new SchedulingAgentCallback() { @@ -564,7 +592,7 @@ public synchronized CompletableFuture stopProcessor(final ProcessorNode pr final LifecycleState lifecycleState = getLifecycleState(procNode, false, false); final StandardProcessContext processContext = new StandardProcessContext(procNode, getControllerServiceProvider(), - getStateManager(procNode), lifecycleState::isTerminated, flowController); + getStateManager(procNode), lifecycleState::isTerminated, nodeTypeProvider); LOG.info("Stopping {}", procNode); return procNode.stop(this, this.componentLifeCycleThreadPool, processContext, getSchedulingAgent(procNode), lifecycleState, lifecycleMethods); @@ -597,7 +625,7 @@ public synchronized void terminateProcessor(final ProcessorNode procNode) { try { final Set additionalUrls = procNode.getAdditionalClasspathResources(procNode.getPropertyDescriptors()); - flowController.getReloadComponent().reload(procNode, procNode.getCanonicalClassName(), procNode.getBundleCoordinate(), additionalUrls); + reloadComponent.reload(procNode, procNode.getCanonicalClassName(), procNode.getBundleCoordinate(), additionalUrls); } catch (final ProcessorInstantiationException e) { // This shouldn't happen because we already have been able to instantiate the processor before LOG.error("Failed to replace instance of Processor for {} when terminating Processor", procNode); @@ -741,7 +769,7 @@ private synchronized void stopConnectable(final Connectable connectable) { if (!state.isScheduled() && state.getActiveThreadCount() == 0 && state.mustCallOnStoppedMethods()) { final StateManager stateManager = (connectable instanceof ProcessorNode) ? getStateManager((ProcessorNode) connectable) : getStateManager(connectable.getIdentifier()); final ConnectableProcessContext processContext = new ConnectableProcessContext(connectable, stateManager); - try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(flowController.getExtensionManager(), connectable.getClass(), connectable.getIdentifier())) { + try (final NarCloseable ignored = NarCloseable.withComponentNarLoader(extensionManager, connectable.getClass(), connectable.getIdentifier())) { ReflectionUtils.quietlyInvokeMethodsWithAnnotation(OnStopped.class, connectable, processContext); } } @@ -852,7 +880,18 @@ public CompletableFuture enableControllerService(final ControllerServiceNo return enableControllerService(service, true); } + @Override + public CompletableFuture enableControllerService(final ControllerServiceNode service, final ConfigurationContext configurationContext) { + return enableControllerService(service, true, configurationContext); + } + private CompletableFuture enableControllerService(final ControllerServiceNode service, final boolean completeFutureExceptionally) { + return enableControllerService(service, completeFutureExceptionally, null); + } + + private CompletableFuture enableControllerService(final ControllerServiceNode service, final boolean completeFutureExceptionally, + final ConfigurationContext configurationContext) { + if (service.isActive()) { LOG.debug("{} is already active, so not enabling it again", service); return CompletableFuture.completedFuture(null); @@ -860,15 +899,18 @@ private CompletableFuture enableControllerService(final ControllerServiceN LOG.info("Enabling {}", service); + service.reloadAdditionalResourcesIfNecessary(); + final List> futures = new ArrayList<>(); final List dependentServices = service.getRequiredControllerServices(); for (final ControllerServiceNode dependentService : dependentServices) { // Enable Controller Service but if it fails, do not complete the future Exceptionally. This allows us to wait up until the // timeout for the service to enable, even if it needs to retry in order to do so. - futures.add(enableControllerService(dependentService, completeFutureExceptionally)); + // Note: dependent services always use their own configuration, not the provided configurationContext + futures.add(enableControllerService(dependentService, completeFutureExceptionally, null)); } - futures.add(service.enable(this.componentLifeCycleThreadPool, this.administrativeYieldMillis, completeFutureExceptionally)); + futures.add(service.enable(this.componentLifeCycleThreadPool, this.administrativeYieldMillis, completeFutureExceptionally, configurationContext)); return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); } @@ -890,6 +932,23 @@ public CompletableFuture disableControllerService(final ControllerServiceN } } + @Override + public CompletableFuture startConnector(final ConnectorNode connectorNode) { + // TODO: Implement + throw new UnsupportedOperationException(); + } + + @Override + public CompletableFuture stopConnector(final ConnectorNode connectorNode) { + // TODO: Implement + throw new UnsupportedOperationException(); + } + + @Override + public void onConnectorRemoved(final ConnectorNode connectorNode) { + lifecycleStateManager.removeLifecycleState(connectorNode.getIdentifier()); + } + private CompletableFuture disableControllerServiceWithStandaloneThreadPool(final ControllerServiceNode service) { final ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); try { diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedDataflowMapper.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedDataflowMapper.java index d55e8d6c620d..550cfbe25f80 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedDataflowMapper.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedDataflowMapper.java @@ -17,6 +17,7 @@ package org.apache.nifi.controller.serialization; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.connectable.Port; import org.apache.nifi.controller.FlowAnalysisRuleNode; import org.apache.nifi.controller.FlowController; @@ -27,6 +28,7 @@ import org.apache.nifi.controller.flow.VersionedFlowEncodingVersion; import org.apache.nifi.controller.service.ControllerServiceNode; import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedConnector; import org.apache.nifi.flow.VersionedControllerService; import org.apache.nifi.flow.VersionedFlowAnalysisRule; import org.apache.nifi.flow.VersionedFlowRegistryClient; @@ -40,8 +42,8 @@ import org.apache.nifi.registry.flow.FlowRegistryClientNode; import org.apache.nifi.registry.flow.mapping.ComponentIdLookup; import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; import org.apache.nifi.registry.flow.mapping.SensitiveValueEncryptor; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; import java.util.ArrayList; @@ -52,7 +54,7 @@ public class VersionedDataflowMapper { private static final VersionedFlowEncodingVersion ENCODING_VERSION = new VersionedFlowEncodingVersion(2, 0); private final FlowController flowController; - private final NiFiRegistryFlowMapper flowMapper; + private final VersionedComponentFlowMapper flowMapper; private final ScheduledStateLookup stateLookup; public VersionedDataflowMapper(final FlowController flowController, final ExtensionManager extensionManager, final SensitiveValueEncryptor encryptor, final ScheduledStateLookup stateLookup) { @@ -73,7 +75,7 @@ public VersionedDataflowMapper(final FlowController flowController, final Extens .mapAssetReferences(true) .build(); - flowMapper = new NiFiRegistryFlowMapper(extensionManager, mappingOptions); + flowMapper = new VersionedComponentFlowMapper(extensionManager, mappingOptions); } public VersionedDataflow createMapping() { @@ -86,11 +88,23 @@ public VersionedDataflow createMapping() { dataflow.setReportingTasks(mapReportingTasks()); dataflow.setFlowAnalysisRules(mapFlowAnalysisRules()); dataflow.setParameterProviders(mapParameterProviders()); + dataflow.setConnectors(mapConnectors()); dataflow.setRootGroup(mapRootGroup()); return dataflow; } + private List mapConnectors() { + final List connectors = new ArrayList<>(); + + for (final ConnectorNode connectorNode : flowController.getConnectorRepository().getConnectors()) { + final VersionedConnector versionedConnector = flowMapper.mapConnector(connectorNode); + connectors.add(versionedConnector); + } + + return connectors; + } + private List mapControllerServices() { final List controllerServices = new ArrayList<>(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java index e0a7ce607fd4..13eb54aec787 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java @@ -26,6 +26,15 @@ import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.cluster.protocol.DataFlow; import org.apache.nifi.cluster.protocol.StandardDataFlow; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConnectorConfiguration; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.NamedStepConfiguration; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StringLiteralValue; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Position; @@ -55,6 +64,9 @@ import org.apache.nifi.flow.ScheduledState; import org.apache.nifi.flow.VersionedAsset; import org.apache.nifi.flow.VersionedComponent; +import org.apache.nifi.flow.VersionedConfigurationStep; +import org.apache.nifi.flow.VersionedConnector; +import org.apache.nifi.flow.VersionedConnectorValueReference; import org.apache.nifi.flow.VersionedControllerService; import org.apache.nifi.flow.VersionedExternalFlow; import org.apache.nifi.flow.VersionedFlowAnalysisRule; @@ -412,9 +424,15 @@ private void synchronizeFlow(final FlowController controller, final DataFlow exi versionedExternalFlow.setParameterContexts(versionedParameterContextMap); versionedExternalFlow.setFlowContents(versionedFlow.getRootGroup()); + // Inherit Parameter Providers and Connectors first. Because Connectors are a bit different, in that updates could result in Exceptions being thrown, + // due to the fact that they manipulate the flow, and changes can be aborted, we handle them first. This way, if there's any Exception, + // we can fail before updating parts of the flow that are not managed by Connectors. + // Because Connectors may depend on Parameter Providers, we need to ensure that we inherit Parameter Providers first. + inheritParameterProviders(controller, versionedFlow, affectedComponentSet); + inheritConnectors(controller, versionedFlow); + // Inherit controller-level components. inheritControllerServices(controller, versionedFlow, affectedComponentSet); - inheritParameterProviders(controller, versionedFlow, affectedComponentSet); inheritParameterContexts(controller, versionedFlow); inheritReportingTasks(controller, versionedFlow, affectedComponentSet); inheritFlowAnalysisRules(controller, versionedFlow, affectedComponentSet); @@ -1023,6 +1041,156 @@ private Set getAssetIds(final Parameter parameter) { .collect(Collectors.toSet()); } + private void inheritConnectors(final FlowController flowController, final VersionedDataflow dataflow) { + // TODO: We need to delete any Connectors that are no longer part of the flow. + // This means we need to drain the Connector first, then stop it, then delete it. If unable to drain, we must fail... + // perhaps we need a DRAINING state? Or do we just delete it and drop the data? + final ConnectorRepository connectorRepository = flowController.getConnectorRepository(); + + final Set proposedConnectorIds = new HashSet<>(); + if (dataflow.getConnectors() != null) { + for (final VersionedConnector versionedConnector : dataflow.getConnectors()) { + proposedConnectorIds.add(versionedConnector.getInstanceIdentifier()); + + final ConnectorNode existingConnector = connectorRepository.getConnector(versionedConnector.getInstanceIdentifier()); + if (existingConnector == null) { + logger.info("Connector {} of type {} with name {} is not in the current flow. Will add Connector.", + versionedConnector.getInstanceIdentifier(), versionedConnector.getType(), versionedConnector.getName()); + + addConnector(versionedConnector, connectorRepository, flowController.getFlowManager()); + } else if (isConnectorConfigurationUpdated(existingConnector, versionedConnector)) { + logger.info("{} configuration has changed, updating configuration", existingConnector); + updateConnector(versionedConnector, connectorRepository); + } else { + logger.debug("{} configuration is up to date, no update necessary", existingConnector); + } + } + } + + for (final ConnectorNode existingConnector : connectorRepository.getConnectors()) { + if (!proposedConnectorIds.contains(existingConnector.getIdentifier())) { + logger.info("Connector {} is no longer part of the proposed flow. Will remove Connector.", existingConnector); + connectorRepository.stopConnector(existingConnector); + connectorRepository.removeConnector(existingConnector.getIdentifier()); + } + } + } + + private boolean isConnectorConfigurationUpdated(final ConnectorNode existingConnector, final VersionedConnector versionedConnector) { + final ConnectorConfiguration activeConfig = existingConnector.getActiveFlowContext().getConfigurationContext().toConnectorConfiguration(); + final List versionedActiveConfig = versionedConnector.getActiveFlowConfiguration(); + final boolean activeContextChanged = isConnectorConfigurationUpdated(activeConfig, versionedActiveConfig); + + final ConnectorConfiguration workingConfig = existingConnector.getWorkingFlowContext().getConfigurationContext().toConnectorConfiguration(); + final List versionedWorkingConfig = versionedConnector.getWorkingFlowConfiguration(); + final boolean workingContextChanged = isConnectorConfigurationUpdated(workingConfig, versionedWorkingConfig); + + return activeContextChanged || workingContextChanged; + } + + private boolean isConnectorConfigurationUpdated(final ConnectorConfiguration existingConfiguration, final List versionedConfigurationSteps) { + if (versionedConfigurationSteps == null || versionedConfigurationSteps.isEmpty()) { + return existingConfiguration != null && !existingConfiguration.getNamedStepConfigurations().isEmpty(); + } + + final Set existingStepConfigurations = existingConfiguration.getNamedStepConfigurations(); + if (existingStepConfigurations.size() != versionedConfigurationSteps.size()) { + return true; + } + + final Map existingStepsByName = existingStepConfigurations.stream() + .collect(Collectors.toMap(NamedStepConfiguration::stepName, Function.identity())); + + for (final VersionedConfigurationStep versionedStep : versionedConfigurationSteps) { + final NamedStepConfiguration existingStep = existingStepsByName.get(versionedStep.getName()); + if (existingStep == null) { + return true; + } + + if (isConfigurationStepUpdated(existingStep, versionedStep)) { + return true; + } + } + + return false; + } + + private boolean isConfigurationStepUpdated(final NamedStepConfiguration existingStep, final VersionedConfigurationStep versionedStep) { + final Map existingProperties = existingStep.configuration().getPropertyValues(); + + final Map versionedProperties = versionedStep.getProperties(); + if (versionedProperties == null || versionedProperties.isEmpty()) { + return existingProperties != null && !existingProperties.isEmpty(); + } + + if (existingProperties == null || existingProperties.size() != versionedProperties.size()) { + return true; + } + + for (final Map.Entry versionedEntry : versionedProperties.entrySet()) { + final String propertyName = versionedEntry.getKey(); + final VersionedConnectorValueReference versionedRef = versionedEntry.getValue(); + final ConnectorValueReference existingRef = existingProperties.get(propertyName); + + if (!equals(versionedRef, existingRef)) { + return true; + } + } + + return false; + } + + private boolean equals(final VersionedConnectorValueReference versionedReference, final ConnectorValueReference existingReference) { + if (versionedReference == null && existingReference == null) { + return true; + } + if (versionedReference == null || existingReference == null) { + return false; + } + + final String versionedValueType = versionedReference.getValueType(); + final String existingValueType = existingReference.getValueType().name(); + if (!Objects.equals(versionedValueType, existingValueType)) { + return false; + } + + return switch (existingReference) { + case StringLiteralValue stringLiteral -> Objects.equals(stringLiteral.getValue(), versionedReference.getValue()); + case AssetReference assetRef -> Objects.equals(assetRef.getAssetIdentifiers(), versionedReference.getAssetIds()); + case SecretReference secretRef -> Objects.equals(secretRef.getProviderId(), versionedReference.getProviderId()) + && Objects.equals(secretRef.getSecretName(), versionedReference.getSecretName()); + }; + } + + private void addConnector(final VersionedConnector versionedConnector, final ConnectorRepository connectorRepository, final FlowManager flowManager) { + final BundleCoordinate coordinate = createBundleCoordinate(extensionManager, versionedConnector.getBundle(), versionedConnector.getType()); + final ConnectorNode connectorNode = flowManager.createConnector(versionedConnector.getType(), versionedConnector.getInstanceIdentifier(), coordinate, false, true); + connectorRepository.restoreConnector(connectorNode); + updateConnector(versionedConnector, connectorRepository); + } + + + private void updateConnector(final VersionedConnector versionedConnector, final ConnectorRepository connectorRepository) { + final ConnectorNode connectorNode = connectorRepository.getConnector(versionedConnector.getInstanceIdentifier()); + + connectorRepository.updateConnector(connectorNode, versionedConnector.getName()); + + try { + final List activeFlowConfig = versionedConnector.getActiveFlowConfiguration(); + final List workingFlowConfig = versionedConnector.getWorkingFlowConfiguration(); + connectorRepository.inheritConfiguration(connectorNode, activeFlowConfig, workingFlowConfig, versionedConnector.getBundle()); + + final ScheduledState desiredState = versionedConnector.getScheduledState(); + if (desiredState == ScheduledState.RUNNING) { + connectorRepository.startConnector(connectorNode); + } else if (desiredState == ScheduledState.ENABLED) { + connectorRepository.stopConnector(connectorNode); + } + } catch (final FlowUpdateException e) { + logger.error("{} failed to inherit configuration", connectorNode, e); + } + } + private void inheritControllerServices(final FlowController controller, final VersionedDataflow dataflow, final AffectedComponentSet affectedComponentSet) { final FlowManager flowManager = controller.getFlowManager(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedReportingTaskSnapshotMapper.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedReportingTaskSnapshotMapper.java index fb00f48da651..dbe4e7a12f62 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedReportingTaskSnapshotMapper.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedReportingTaskSnapshotMapper.java @@ -23,7 +23,7 @@ import org.apache.nifi.flow.VersionedReportingTask; import org.apache.nifi.flow.VersionedReportingTaskSnapshot; import org.apache.nifi.nar.ExtensionManager; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import java.util.ArrayList; import java.util.Collections; @@ -32,11 +32,11 @@ public class VersionedReportingTaskSnapshotMapper { - private final NiFiRegistryFlowMapper flowMapper; + private final VersionedComponentFlowMapper flowMapper; private final ControllerServiceProvider controllerServiceProvider; public VersionedReportingTaskSnapshotMapper(final ExtensionManager extensionManager, final ControllerServiceProvider controllerServiceProvider) { - this.flowMapper = new NiFiRegistryFlowMapper(extensionManager); + this.flowMapper = new VersionedComponentFlowMapper(extensionManager); this.controllerServiceProvider = controllerServiceProvider; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/StatelessFlowTask.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/StatelessFlowTask.java index b1276eb52b4b..b34a8cd1e84a 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/StatelessFlowTask.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/StatelessFlowTask.java @@ -66,6 +66,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; import java.util.Set; import java.util.concurrent.TimeUnit; import java.util.function.Predicate; @@ -227,6 +228,8 @@ public synchronized void trigger() { fail(successfulInvocations, statelessProvRepo, e); } + updateFlowFileActivity(statelessProvRepo); + logger.debug("Acknowledging FlowFiles from {} invocations", allInvocations.size()); for (final Invocation invocation : allInvocations) { for (final PolledFlowFile polledFlowFile : invocation.getPolledFlowFiles()) { @@ -618,6 +621,72 @@ void updateEventRepository(final List invocations) { } } + + /** + * Updates the Stateless Group Node's FlowFileActivity. The latest activity time is obtained from the stateless flow's root group, + * which is updated by processors within the flow when their sessions commit. Transfer counts are computed from provenance events + * generated during the stateless flow execution, mirroring the behavior of StandardProcessSession.updateTransferCounts. + * + * @param statelessProvRepo the provenance event repository used during stateless flow execution + */ + void updateFlowFileActivity(final ProvenanceEventRepository statelessProvRepo) { + final OptionalLong latestActivityTime = flow.getLatestActivityTime(); + if (latestActivityTime.isPresent()) { + statelessGroupNode.getFlowFileActivity().updateLatestActivityTime(); + } + + updateTransferCounts(statelessProvRepo); + } + + private void updateTransferCounts(final ProvenanceEventRepository statelessProvRepo) { + int receivedCount = 0; + long receivedBytes = 0L; + int sentCount = 0; + long sentBytes = 0L; + + long firstProvEventId = 0; + while (true) { + try { + final List events = statelessProvRepo.getEvents(firstProvEventId, 1000); + if (events.isEmpty()) { + break; + } + + for (final ProvenanceEventRecord event : events) { + final ProvenanceEventType eventType = event.getEventType(); + switch (eventType) { + case RECEIVE: + case CREATE: + receivedCount++; + receivedBytes += event.getFileSize(); + break; + case FETCH: + receivedBytes += event.getFileSize(); + break; + case SEND: + sentCount++; + sentBytes += event.getFileSize(); + break; + default: + break; + } + } + + if (events.size() < 1000) { + break; + } + firstProvEventId += 1000; + } catch (final IOException e) { + logger.warn("Failed to obtain Provenance Events for FlowFile Activity tracking", e); + break; + } + } + + if (receivedCount > 0 || receivedBytes > 0L || sentCount > 0 || sentBytes > 0L) { + statelessGroupNode.getFlowFileActivity().updateTransferCounts(receivedCount, receivedBytes, sentCount, sentBytes); + } + } + private void expireRecords(final FlowFileQueue sourceQueue, final Set expiredRecords) throws IOException { if (expiredRecords.isEmpty()) { return; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/BootstrapDiagnosticsFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/BootstrapDiagnosticsFactory.java index 994aafddcbef..9d4049e5b9b9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/BootstrapDiagnosticsFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/BootstrapDiagnosticsFactory.java @@ -25,6 +25,7 @@ import org.apache.nifi.diagnostics.ThreadDumpTask; import org.apache.nifi.diagnostics.bootstrap.tasks.ClusterDiagnosticTask; import org.apache.nifi.diagnostics.bootstrap.tasks.ComponentCountTask; +import org.apache.nifi.diagnostics.bootstrap.tasks.ConnectionDiagnosticTask; import org.apache.nifi.diagnostics.bootstrap.tasks.ContentRepositoryScanTask; import org.apache.nifi.diagnostics.bootstrap.tasks.DataValveDiagnosticsTask; import org.apache.nifi.diagnostics.bootstrap.tasks.DiagnosticAnalysisTask; @@ -82,6 +83,7 @@ public List getDiagnosticTasks() { tasks.add(new ClusterDiagnosticTask(flowController)); tasks.add(new GarbageCollectionDiagnosticTask(flowController)); tasks.add(new MemoryPoolPeakUsageTask()); + tasks.add(new ConnectionDiagnosticTask(flowController)); tasks.add(new RepositoryDiagnosticTask(flowController)); tasks.add(new ComponentCountTask(flowController)); tasks.add(new NiFiPropertiesDiagnosticTask(nifiProperties)); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/tasks/ConnectionDiagnosticTask.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/tasks/ConnectionDiagnosticTask.java new file mode 100644 index 000000000000..a7294307ac4b --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/diagnostics/bootstrap/tasks/ConnectionDiagnosticTask.java @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.diagnostics.bootstrap.tasks; + +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.queue.FlowFileQueue; +import org.apache.nifi.controller.queue.LoadBalanceCompression; +import org.apache.nifi.controller.queue.LoadBalanceStrategy; +import org.apache.nifi.controller.queue.LocalQueuePartitionDiagnostics; +import org.apache.nifi.controller.queue.QueueDiagnostics; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.queue.RemoteQueuePartitionDiagnostics; +import org.apache.nifi.diagnostics.DiagnosticTask; +import org.apache.nifi.diagnostics.DiagnosticsDumpElement; +import org.apache.nifi.diagnostics.StandardDiagnosticsDumpElement; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.util.FormatUtils; + +import java.text.NumberFormat; +import java.util.ArrayList; +import java.util.List; + +public class ConnectionDiagnosticTask implements DiagnosticTask { + private static final String TABLE_FORMAT = "| %-40s | %-25s | %-25s | %20s | %20s | %20s | %-40s | %15s | %20s | %-19s | %-45s |"; + private static final String TABLE_HEADER = String.format(TABLE_FORMAT, + "Connection ID", "Source", "Destination", "Queued", "Active", "Swap Queue", "Swap Files", "Unacknowledged", "Penalized", "FlowFile Expiration", "Load Balancing"); + private static final String TABLE_SEPARATOR = "+" + "-".repeat(42) + "+" + "-".repeat(27) + "+" + "-".repeat(27) + "+" + + "-".repeat(22) + "+" + "-".repeat(22) + "+" + "-".repeat(22) + "+" + "-".repeat(42) + "+" + "-".repeat(17) + "+" + "-".repeat(22) + "+" + "-".repeat(21) + "+" + "-".repeat(47) + "+"; + + private final FlowController flowController; + + public ConnectionDiagnosticTask(final FlowController flowController) { + this.flowController = flowController; + } + + @Override + public DiagnosticsDumpElement captureDump(final boolean verbose) { + final List details = new ArrayList<>(); + + final ProcessGroup rootGroup = flowController.getFlowManager().getRootGroup(); + details.add("=== Connections in Primary Flow ==="); + captureConnectionDiagnostics(rootGroup, details); + + details.add(""); + final ConnectorRepository connectorRepository = flowController.getConnectorRepository(); + final List connectors = connectorRepository != null ? connectorRepository.getConnectors() : List.of(); + if (connectors.isEmpty()) { + details.add("This instance has no Connectors."); + } else { + details.add("=== Connections in Connectors ==="); + for (final ConnectorNode connector : connectors) { + details.add(""); + details.add("Connector: " + connector.getName() + " (ID: " + connector.getIdentifier() + ", State: " + connector.getCurrentState() + ")"); + final ProcessGroup managedGroup = connector.getActiveFlowContext().getManagedProcessGroup(); + captureConnectionDiagnostics(managedGroup, details); + } + } + + return new StandardDiagnosticsDumpElement("Connection Diagnostics", details); + } + + private void captureConnectionDiagnostics(final ProcessGroup group, final List details) { + final List connections = group.findAllConnections(); + + if (connections.isEmpty()) { + details.add("No connections"); + return; + } + + details.add(TABLE_SEPARATOR); + details.add(TABLE_HEADER); + details.add(TABLE_SEPARATOR); + + for (final Connection connection : connections) { + captureConnectionRow(connection, details); + capturePartitionRows(connection, details); + } + + details.add(TABLE_SEPARATOR); + details.add("Total: " + connections.size() + " connections"); + } + + private void captureConnectionRow(final Connection connection, final List details) { + final FlowFileQueue queue = connection.getFlowFileQueue(); + final String connectionId = connection.getIdentifier(); + final String sourceName = truncate(connection.getSource().getName(), 25); + final String destName = truncate(connection.getDestination().getName(), 25); + + final QueueSize totalSize = queue.size(); + final String queued = formatQueueSize(totalSize); + final String loadBalancing = formatLoadBalancing(queue); + + final String flowFileExpiration = queue.getFlowFileExpiration(); + final String expirationDisplay = "0 sec".equals(flowFileExpiration) ? "No Expiration" : flowFileExpiration; + + // Gather totals from all partitions + String active = "-"; + String swapQueue = "-"; + String swapFiles = "-"; + String unack = "-"; + String penalized = "0"; + + final QueueDiagnostics diagnostics = queue.getQueueDiagnostics(); + if (diagnostics != null) { + int totalActive = 0; + long totalActiveBytes = 0; + int totalSwapQueue = 0; + long totalSwapQueueBytes = 0; + int totalSwapFileCount = 0; + int totalSwapFlowFiles = 0; + long totalSwapBytes = 0; + int totalUnack = 0; + long totalUnackBytes = 0; + int totalPenalizedCount = 0; + long totalPenalizedBytes = 0; + + final LocalQueuePartitionDiagnostics localDiagnostics = diagnostics.getLocalQueuePartitionDiagnostics(); + if (localDiagnostics != null) { + totalActive += localDiagnostics.getActiveQueueSize().getObjectCount(); + totalActiveBytes += localDiagnostics.getActiveQueueSize().getByteCount(); + totalSwapQueue += localDiagnostics.getSwapQueueSize().getObjectCount(); + totalSwapQueueBytes += localDiagnostics.getSwapQueueSize().getByteCount(); + totalSwapFileCount += localDiagnostics.getSwapFileCount(); + totalSwapFlowFiles += localDiagnostics.getTotalSwapFileQueueSize().getObjectCount(); + totalSwapBytes += localDiagnostics.getTotalSwapFileQueueSize().getByteCount(); + totalUnack += localDiagnostics.getUnacknowledgedQueueSize().getObjectCount(); + totalUnackBytes += localDiagnostics.getUnacknowledgedQueueSize().getByteCount(); + totalPenalizedCount += localDiagnostics.getPenalizedQueueSize().getObjectCount(); + totalPenalizedBytes += localDiagnostics.getPenalizedQueueSize().getByteCount(); + } + + final List remotePartitions = diagnostics.getRemoteQueuePartitionDiagnostics(); + if (remotePartitions != null) { + for (final RemoteQueuePartitionDiagnostics remoteDiagnostics : remotePartitions) { + totalActive += remoteDiagnostics.getActiveQueueSize().getObjectCount(); + totalActiveBytes += remoteDiagnostics.getActiveQueueSize().getByteCount(); + totalSwapQueue += remoteDiagnostics.getSwapQueueSize().getObjectCount(); + totalSwapQueueBytes += remoteDiagnostics.getSwapQueueSize().getByteCount(); + totalSwapFileCount += remoteDiagnostics.getSwapFileCount(); + totalUnack += remoteDiagnostics.getUnacknowledgedQueueSize().getObjectCount(); + totalUnackBytes += remoteDiagnostics.getUnacknowledgedQueueSize().getByteCount(); + } + } + + active = formatQueueSize(new QueueSize(totalActive, totalActiveBytes)); + swapQueue = formatQueueSize(new QueueSize(totalSwapQueue, totalSwapQueueBytes)); + swapFiles = formatSwapFiles(totalSwapFileCount, totalSwapFlowFiles, totalSwapBytes); + unack = formatQueueSize(new QueueSize(totalUnack, totalUnackBytes)); + penalized = formatQueueSize(new QueueSize(totalPenalizedCount, totalPenalizedBytes)); + } + + details.add(String.format(TABLE_FORMAT, connectionId, sourceName, destName, queued, active, swapQueue, swapFiles, unack, penalized, expirationDisplay, loadBalancing)); + } + + private void capturePartitionRows(final Connection connection, final List details) { + final FlowFileQueue queue = connection.getFlowFileQueue(); + + // Only show partition rows if load balancing is enabled + if (queue.getLoadBalanceStrategy() == LoadBalanceStrategy.DO_NOT_LOAD_BALANCE) { + return; + } + + final QueueDiagnostics diagnostics = queue.getQueueDiagnostics(); + if (diagnostics == null) { + return; + } + + final LocalQueuePartitionDiagnostics localDiagnostics = diagnostics.getLocalQueuePartitionDiagnostics(); + if (localDiagnostics != null) { + final String active = formatQueueSize(localDiagnostics.getActiveQueueSize()); + final String swapQueue = formatQueueSize(localDiagnostics.getSwapQueueSize()); + final int swapFileCount = localDiagnostics.getSwapFileCount(); + final QueueSize swapFileQueueSize = localDiagnostics.getTotalSwapFileQueueSize(); + final String swapFiles = formatSwapFiles(swapFileCount, swapFileQueueSize.getObjectCount(), swapFileQueueSize.getByteCount()); + final QueueSize unacknowledgedSize = localDiagnostics.getUnacknowledgedQueueSize(); + final String unack = unacknowledgedSize.getObjectCount() > 0 ? formatQueueSize(unacknowledgedSize) : "-"; + final String penalized = formatQueueSize(localDiagnostics.getPenalizedQueueSize()); + + details.add(String.format(TABLE_FORMAT, " - Local Partition", "", "", "", active, swapQueue, swapFiles, unack, penalized, "", "")); + } + + final List remotePartitions = diagnostics.getRemoteQueuePartitionDiagnostics(); + if (remotePartitions != null && !remotePartitions.isEmpty()) { + for (final RemoteQueuePartitionDiagnostics remoteDiagnostics : remotePartitions) { + final String nodeId = remoteDiagnostics.getNodeIdentifier(); + final String active = formatQueueSize(remoteDiagnostics.getActiveQueueSize()); + final String swapQueue = formatQueueSize(remoteDiagnostics.getSwapQueueSize()); + final String swapFiles = String.valueOf(remoteDiagnostics.getSwapFileCount()); + final QueueSize unacknowledgedSize = remoteDiagnostics.getUnacknowledgedQueueSize(); + final String unack = unacknowledgedSize.getObjectCount() > 0 ? formatQueueSize(unacknowledgedSize) : "-"; + + details.add(String.format(TABLE_FORMAT, " - Remote Partition - " + nodeId, "", "", "", active, swapQueue, swapFiles, unack, "-", "", "")); + } + } + } + + private String formatLoadBalancing(final FlowFileQueue queue) { + final LoadBalanceStrategy strategy = queue.getLoadBalanceStrategy(); + if (strategy == LoadBalanceStrategy.DO_NOT_LOAD_BALANCE) { + return "None"; + } + + final StringBuilder builder = new StringBuilder(); + builder.append(strategy.name()); + + if (strategy == LoadBalanceStrategy.PARTITION_BY_ATTRIBUTE) { + final String partitioningAttribute = queue.getPartitioningAttribute(); + if (partitioningAttribute != null && !partitioningAttribute.isEmpty()) { + builder.append(" (").append(partitioningAttribute).append(")"); + } + } + + final LoadBalanceCompression compression = queue.getLoadBalanceCompression(); + if (compression != LoadBalanceCompression.DO_NOT_COMPRESS) { + builder.append(", ").append(compression.name()); + } + + return builder.toString(); + } + + private String formatQueueSize(final QueueSize queueSize) { + if (queueSize.getObjectCount() == 0) { + return "0"; + } + final String formattedCount = NumberFormat.getIntegerInstance().format(queueSize.getObjectCount()); + return formattedCount + " (" + FormatUtils.formatDataSize(queueSize.getByteCount()) + ")"; + } + + private String formatSwapFiles(final int fileCount, final int flowFileCount, final long totalBytes) { + if (fileCount == 0) { + return "0"; + } + final String files = fileCount == 1 ? "1 File" : NumberFormat.getIntegerInstance().format(fileCount) + " Files"; + final String flowFiles = NumberFormat.getIntegerInstance().format(flowFileCount) + " FlowFiles"; + final String size = FormatUtils.formatDataSize(totalBytes); + return files + " - " + flowFiles + " (" + size + ")"; + } + + private String truncate(final String value, final int maxLength) { + if (value == null) { + return ""; + } + if (value.length() <= maxLength) { + return value; + } + return value.substring(0, maxLength - 3) + "..."; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/flowanalysis/StandardFlowAnalyzer.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/flowanalysis/StandardFlowAnalyzer.java index 63ae900d5117..ce0beac1ad98 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/flowanalysis/StandardFlowAnalyzer.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/flowanalysis/StandardFlowAnalyzer.java @@ -30,7 +30,7 @@ import org.apache.nifi.flow.VersionedProcessGroup; import org.apache.nifi.flow.VersionedProcessor; import org.apache.nifi.nar.ExtensionManager; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.validation.RuleViolation; import org.apache.nifi.validation.RuleViolationsManager; import org.slf4j.Logger; @@ -89,7 +89,7 @@ public void setFlowAnalysisRequired(boolean flowAnalysisRequired) { public void analyzeProcessor(ProcessorNode processorNode) { logger.debug("Running analysis on {}", processorNode); - final NiFiRegistryFlowMapper mapper = createMapper(); + final VersionedComponentFlowMapper mapper = createMapper(); VersionedProcessor versionedProcessor = mapper.mapProcessor( processorNode, @@ -105,7 +105,7 @@ public void analyzeProcessor(ProcessorNode processorNode) { public void analyzeControllerService(ControllerServiceNode controllerServiceNode) { logger.debug("Running analysis on {}", controllerServiceNode); - final NiFiRegistryFlowMapper mapper = createMapper(); + final VersionedComponentFlowMapper mapper = createMapper(); VersionedControllerService versionedControllerService = mapper.mapControllerService( controllerServiceNode, @@ -266,8 +266,8 @@ private String getDisplayName(VersionedComponent component) { return displayName; } - private NiFiRegistryFlowMapper createMapper() { - NiFiRegistryFlowMapper mapper = FlowAnalysisUtil.createMapper(extensionManager); + private VersionedComponentFlowMapper createMapper() { + VersionedComponentFlowMapper mapper = FlowAnalysisUtil.createMapper(extensionManager); return mapper; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/framework/configuration/FlowControllerConfiguration.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/framework/configuration/FlowControllerConfiguration.java index 88d760b34160..63aff7636a6e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/framework/configuration/FlowControllerConfiguration.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/framework/configuration/FlowControllerConfiguration.java @@ -25,11 +25,13 @@ import org.apache.nifi.asset.AssetSynchronizer; import org.apache.nifi.asset.StandardAssetComponentManager; import org.apache.nifi.asset.StandardAssetSynchronizer; +import org.apache.nifi.asset.StandardConnectorAssetSynchronizer; import org.apache.nifi.authorization.Authorizer; import org.apache.nifi.cluster.coordination.ClusterCoordinator; import org.apache.nifi.cluster.coordination.heartbeat.HeartbeatMonitor; import org.apache.nifi.cluster.protocol.NodeProtocolSender; import org.apache.nifi.cluster.protocol.impl.NodeProtocolSenderListener; +import org.apache.nifi.components.connector.ConnectorRequestReplicator; import org.apache.nifi.components.state.StateManagerProvider; import org.apache.nifi.controller.FlowController; import org.apache.nifi.controller.StandardFlowService; @@ -95,34 +97,21 @@ public class FlowControllerConfiguration { private static final String COMPONENT_METRIC_REPORTER_IMPLEMENTATION = "nifi.component.metric.reporter.implementation"; private NiFiProperties properties; - private ExtensionDiscoveringManager extensionManager; - private AuditService auditService; - private Authorizer authorizer; - private RevisionManager revisionManager; - private LeaderElectionManager leaderElectionManager; - private SSLContext sslContext; - private X509KeyManager keyManager; - private X509TrustManager trustManager; - private StateManagerProvider stateManagerProvider; - private BulletinRepository bulletinRepository; - private NodeProtocolSender nodeProtocolSender; - private NodeProtocolSenderListener nodeProtocolSenderListener; - private HeartbeatMonitor heartbeatMonitor; - private ClusterCoordinator clusterCoordinator; + private ConnectorRequestReplicator connectorRequestReplicator; @Autowired public void setProperties(final NiFiProperties properties) { @@ -164,6 +153,11 @@ public void setBulletinRepository(final BulletinRepository bulletinRepository) { this.bulletinRepository = bulletinRepository; } + @Autowired(required = false) + public void setConnectorRequestReplicator(final ConnectorRequestReplicator connectorRequestReplicator) { + this.connectorRequestReplicator = connectorRequestReplicator; + } + @Autowired(required = false) public void setSslContext(final SSLContext sslContext) { this.sslContext = sslContext; @@ -200,6 +194,7 @@ public void setClusterCoordinator(final ClusterCoordinator clusterCoordinator) { this.clusterCoordinator = clusterCoordinator; } + /** * Flow Controller implementation depends on cluster configuration * @@ -223,7 +218,8 @@ public FlowController flowController() throws Exception { extensionManager, statusHistoryRepository(), ruleViolationsManager(), - stateManagerProvider + stateManagerProvider, + connectorRequestReplicator ); } else { flowController = FlowController.createClusteredInstance( @@ -243,7 +239,8 @@ public FlowController flowController() throws Exception { revisionManager, statusHistoryRepository(), ruleViolationsManager(), - stateManagerProvider + stateManagerProvider, + connectorRequestReplicator ); } @@ -266,7 +263,8 @@ public FlowService flowService(@Autowired final NarManager narManager) throws Ex properties, revisionManager, narManager, - assetSynchronizer(), + parameterContextAssetSynchronizer(), + connectorAssetSynchronizer(), authorizer ); } else { @@ -277,7 +275,8 @@ public FlowService flowService(@Autowired final NarManager narManager) throws Ex clusterCoordinator, revisionManager, narManager, - assetSynchronizer(), + parameterContextAssetSynchronizer(), + connectorAssetSynchronizer(), authorizer ); } @@ -466,7 +465,7 @@ public AssetManager assetManager() throws Exception { } /** - * Asset Synchronizer depends on ClusterCoordinator, WebClientService, and NiFiProperties + * Parameter Conext Asset Synchronizer depends on ClusterCoordinator, WebClientService, and NiFiProperties * * @return Asset Synchronizer */ @@ -475,6 +474,21 @@ public AssetSynchronizer assetSynchronizer() throws Exception { return new StandardAssetSynchronizer(flowController(), clusterCoordinator, webClientService(), properties, affectedComponentManager()); } + @Bean + public AssetSynchronizer parameterContextAssetSynchronizer() throws Exception { + return new StandardAssetSynchronizer(flowController(), clusterCoordinator, webClientService(), properties, affectedComponentManager()); + } + + /** + * Connector Asset Synchronizer depends on ClusterCoordinator, WebClientService, and NiFiProperties + * + * @return Connector Asset Synchronizer + */ + @Bean + public AssetSynchronizer connectorAssetSynchronizer() throws Exception { + return new StandardConnectorAssetSynchronizer(flowController(), clusterCoordinator, webClientService(), properties); + } + /** * Affected Component Manager depends on FlowController * diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardStatelessGroupNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardStatelessGroupNode.java index a6c076190d34..ac727bdccdb3 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardStatelessGroupNode.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardStatelessGroupNode.java @@ -23,8 +23,10 @@ import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.state.StateManagerProvider; import org.apache.nifi.components.state.StatelessStateManagerProvider; +import org.apache.nifi.connectable.ConnectableFlowFileActivity; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Port; import org.apache.nifi.connectable.Position; import org.apache.nifi.controller.BackoffMechanism; @@ -120,6 +122,8 @@ public class StandardStatelessGroupNode implements StatelessGroupNode { private volatile long schedulingNanos = 1L; private ComponentLog logger; + private final ConnectableFlowFileActivity flowFileActivity = new ConnectableFlowFileActivity(); + private StandardStatelessGroupNode(final Builder builder) { this.processGroup = builder.rootGroup; this.controllerServiceProvider = builder.controllerServiceProvider; @@ -151,6 +155,7 @@ public void start(final ScheduledExecutorService executor, final SchedulingAgent return; } + flowFileActivity.reset(); currentState = ScheduledState.STARTING; logger.info("Starting {}", this); } finally { @@ -1036,4 +1041,9 @@ public StatelessGroupNode build() { } } + + @Override + public FlowFileActivity getFlowFileActivity() { + return flowFileActivity; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/manifest/StandardRuntimeManifestService.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/manifest/StandardRuntimeManifestService.java index 1a7f3ef214a1..320f59af99b5 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/manifest/StandardRuntimeManifestService.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/manifest/StandardRuntimeManifestService.java @@ -401,6 +401,51 @@ private Map discoverAdditionalDetails(final BundleDetails bundleDe return additionalDetailsMap; } + @Override + public Map discoverStepDocumentation(final String group, final String artifact, final String version, final String connectorType) { + final BundleCoordinate bundleCoordinate = new BundleCoordinate(group, artifact, version); + final Bundle bundle = extensionManager.getBundle(bundleCoordinate); + + if (bundle == null) { + throw new ResourceNotFoundException("Unable to find bundle [" + bundleCoordinate + "]"); + } + + return discoverStepDocumentation(bundle.getBundleDetails(), connectorType); + } + + private Map discoverStepDocumentation(final BundleDetails bundleDetails, final String connectorType) { + final Map stepDocsMap = new LinkedHashMap<>(); + + final File stepDocsDir = new File(bundleDetails.getWorkingDirectory(), "META-INF/docs/steps/" + connectorType); + if (!stepDocsDir.exists()) { + LOGGER.debug("No steps directory found for [{}] under [{}]", connectorType, bundleDetails.getWorkingDirectory().getAbsolutePath()); + return stepDocsMap; + } + + final File[] stepDocFiles = stepDocsDir.listFiles(); + if (stepDocFiles == null) { + return stepDocsMap; + } + + for (final File stepDocFile : stepDocFiles) { + if (!stepDocFile.isFile() || !stepDocFile.getName().endsWith(".md")) { + LOGGER.debug("Skipping [{}], not a markdown file...", stepDocFile.getAbsolutePath()); + continue; + } + + final String fileName = stepDocFile.getName().substring(0, stepDocFile.getName().length() - 3); + final String stepName = fileNameToStepName(fileName); + stepDocsMap.put(stepName, stepDocFile); + LOGGER.debug("Discovered step documentation for step [{}] at [{}]", stepName, stepDocFile.getAbsolutePath()); + } + + return stepDocsMap; + } + + private String fileNameToStepName(final String fileName) { + return fileName.replace("_", " "); + } + private Map loadAdditionalDetails(final BundleDetails bundleDetails) { final Map additionalDetailsMap = discoverAdditionalDetails(bundleDetails); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/nar/NarInstallTask.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/nar/NarInstallTask.java index 8c2673353d54..ffdb2fa43565 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/nar/NarInstallTask.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/nar/NarInstallTask.java @@ -20,6 +20,7 @@ import org.apache.nifi.bundle.Bundle; import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.bundle.BundleDetails; +import org.apache.nifi.components.connector.Connector; import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.service.ControllerServiceProvider; import org.apache.nifi.flowanalysis.FlowAnalysisRule; @@ -54,7 +55,8 @@ public class NarInstallTask implements Runnable { ReportingTask.class, FlowRegistryClient.class, FlowAnalysisRule.class, - ParameterProvider.class + ParameterProvider.class, + Connector.class ); private final NarNode narNode; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactory.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactory.java index bcc06a33934b..0a2d3387a697 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactory.java @@ -50,7 +50,7 @@ public Authorizable createLocalDataAuthorizable(final String componentId) { final Connectable connectable = flowManager.findConnectable(componentId); if (connectable == null) { // if the component id is not a connectable then consider a connection - final Connection connection = flowManager.getRootGroup().findConnection(componentId); + final Connection connection = flowController.findConnectionIncludingConnectorManaged(componentId); if (connection == null) { throw new ResourceNotFoundException("The component that generated this event is no longer part of the data flow."); @@ -70,7 +70,7 @@ public Authorizable createLocalDataAuthorizable(final String componentId) { public Authorizable createRemoteDataAuthorizable(String remoteGroupPortId) { final DataAuthorizable authorizable; - final RemoteGroupPort remoteGroupPort = flowController.getFlowManager().getRootGroup().findRemoteGroupPort(remoteGroupPortId); + final RemoteGroupPort remoteGroupPort = flowController.findRemoteGroupPortIncludingConnectorManaged(remoteGroupPortId); if (remoteGroupPort == null) { throw new ResourceNotFoundException("The component that generated this event is no longer part of the data flow."); } else { @@ -98,7 +98,7 @@ public Authorizable createProvenanceDataAuthorizable(String componentId) { final Connectable connectable = flowManager.findConnectable(componentId); if (connectable == null) { // if the component id is not a connectable then consider a connection - final Connection connection = flowManager.getRootGroup().findConnection(componentId); + final Connection connection = flowController.findConnectionIncludingConnectorManaged(componentId); if (connection == null) { throw new ResourceNotFoundException("The component that generated this event is no longer part of the data flow."); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository new file mode 100644 index 000000000000..8ce9c5d3ce38 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/META-INF/services/org.apache.nifi.components.connector.ConnectorRepository @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.components.connector.StandardConnectorRepository diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/BlockingConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/BlockingConnector.java new file mode 100644 index 000000000000..75c35154d693 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/BlockingConnector.java @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; + +public class BlockingConnector implements Connector { + private final CountDownLatch startLatch; + private final CountDownLatch stopLatch; + private final CountDownLatch finishUpdateLatch; + + public BlockingConnector(final CountDownLatch startLatch, final CountDownLatch stopLatch, final CountDownLatch finishUpdateLatch) { + this.startLatch = startLatch; + this.stopLatch = stopLatch; + this.finishUpdateLatch = finishUpdateLatch; + } + + + @Override + public void initialize(final ConnectorInitializationContext connectorInitializationContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void start(final FlowContext activeContext) throws FlowUpdateException { + try { + startLatch.await(); + } catch (final InterruptedException e) { + throw new FlowUpdateException(e); + } + } + + @Override + public void stop(final FlowContext activeContext) throws FlowUpdateException { + try { + stopLatch.await(); + } catch (final InterruptedException e) { + throw new FlowUpdateException(e); + } + } + + @Override + public List validate(final FlowContext flowContext, final ConnectorValidationContext connectorValidationContext) { + return List.of(); + } + + @Override + public List validateConfigurationStep(final ConfigurationStep configurationStep, final ConnectorConfigurationContext connectorConfigurationContext, + final ConnectorValidationContext connectorValidationContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void onConfigurationStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public void abortUpdate(final FlowContext workingContext, final Throwable throwable) { + + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + try { + finishUpdateLatch.await(); + } catch (final InterruptedException e) { + throw new FlowUpdateException(e); + } + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List verify(final FlowContext flowContext) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext, final String filter) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext) { + return List.of(); + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + return CompletableFuture.completedFuture(null); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/DynamicAllowableValuesConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/DynamicAllowableValuesConnector.java new file mode 100644 index 000000000000..404a78b9727f --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/DynamicAllowableValuesConnector.java @@ -0,0 +1,142 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.io.File; +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class DynamicAllowableValuesConnector extends AbstractConnector { + static final ConnectorPropertyDescriptor FILE_PATH = new ConnectorPropertyDescriptor.Builder() + .name("File Path") + .description("The path to the file") + .addValidator(new SimpleFileExistsValidator()) + .required(true) + .build(); + + static final ConnectorPropertyGroup FILE_PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("") + .addProperty(FILE_PATH) + .build(); + + static final ConfigurationStep FILE_STEP = new ConfigurationStep.Builder() + .name("File") + .propertyGroups(List.of(FILE_PROPERTY_GROUP)) + .build(); + + static final ConnectorPropertyDescriptor FIRST_PRIMARY_COLOR = new ConnectorPropertyDescriptor.Builder() + .name("First Primary Color") + .description("The first primary color") + .allowableValuesFetchable(true) + .required(true) + .build(); + + static final ConnectorPropertyGroup PRIMARY_COLORS_PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Primary Colors") + .addProperty(FIRST_PRIMARY_COLOR) + .build(); + + static final ConfigurationStep COLORS_STEP = new ConfigurationStep.Builder() + .name("Colors") + .propertyGroups(List.of(PRIMARY_COLORS_PROPERTY_GROUP)) + .dependsOn(FILE_STEP, FILE_PATH) + .build(); + + private static final List CONFIGURATION_STEPS = List.of(FILE_STEP, COLORS_STEP); + + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public List getConfigurationSteps() { + return CONFIGURATION_STEPS; + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final VersionedExternalFlow externalFlow = VersionedFlowUtils.loadFlowFromResource("flows/choose-color.json"); + final VersionedProcessGroup rootGroup = externalFlow.getFlowContents(); + final VersionedProcessor processor = rootGroup.getProcessors().iterator().next(); + processor.setProperties(Map.of("File", workingContext.getConfigurationContext().getProperty(FILE_STEP, FILE_PATH).getValue())); + + getInitializationContext().updateFlow(activeContext, externalFlow); + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext) { + if ("Colors".equals(stepName) && "First Primary Color".equals(propertyName)) { + final Set processorFacades = flowContext.getRootGroup().getProcessors(); + if (!processorFacades.isEmpty()) { + final ProcessorFacade processorFacade = processorFacades.iterator().next(); + + try { + @SuppressWarnings("unchecked") + final List fileValues = (List) processorFacade.invokeConnectorMethod("getFileValues", Map.of()); + + return fileValues.stream() + .map(AllowableValue::new) + .map(DescribedValue.class::cast) + .toList(); + } catch (final InvocationFailedException e) { + throw new RuntimeException("Failed to fetch allowable values from connector.", e); + } + } + } + + return super.fetchAllowableValues(stepName, propertyName, flowContext); + } + + + public static class SimpleFileExistsValidator implements Validator { + @Override + public ValidationResult validate(final String subject, final String input, final ValidationContext validationContext) { + final File file = new File(input); + if (file.exists()) { + return new ValidationResult.Builder().subject(subject).input(input).valid(true).build(); + } + + return new ValidationResult.Builder().subject(subject).input(input).valid(false).explanation("File does not exist.").build(); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/DynamicFlowConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/DynamicFlowConnector.java new file mode 100644 index 000000000000..bf8148214393 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/DynamicFlowConnector.java @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.processors.CreateDummyFlowFile; +import org.apache.nifi.components.connector.processors.DuplicateFlowFile; +import org.apache.nifi.components.connector.processors.LogFlowFileContents; +import org.apache.nifi.components.connector.processors.OverwriteFlowFile; +import org.apache.nifi.components.connector.services.impl.StandardCounterService; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.ConnectableComponent; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class DynamicFlowConnector extends AbstractConnector { + private volatile boolean initialized = false; + + static final ConnectorPropertyDescriptor SOURCE_TEXT = new ConnectorPropertyDescriptor.Builder() + .name("Source Text") + .type(PropertyType.STRING) + .addValidator(Validator.VALID) + .required(true) + .defaultValue("Hello World") + .build(); + + static final ConnectorPropertyDescriptor COUNT_FLOWFILES = new ConnectorPropertyDescriptor.Builder() + .name("Count FlowFiles") + .description("If true, the Counter Service will be used to count the number of FlowFiles created") + .type(PropertyType.BOOLEAN) + .allowableValues("true", "false") + .required(true) + .defaultValue("false") + .build(); + + static final ConnectorPropertyDescriptor NUM_COPIES = new ConnectorPropertyDescriptor.Builder() + .name("Number of Copies") + .type(PropertyType.INTEGER) + .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR) + .required(true) + .defaultValue("1") + .build(); + + static final ConnectorPropertyDescriptor LOG_FLOWFILE_CONTENTS = new ConnectorPropertyDescriptor.Builder() + .name("Log FlowFile Contents") + .type(PropertyType.BOOLEAN) + .allowableValues("true", "false") + .required(true) + .defaultValue("false") + .build(); + + private static final ConnectorPropertyGroup SOURCE_GROUP = new ConnectorPropertyGroup.Builder() + .name("Source Settings") + .description("Settings for the source of FlowFiles") + .addProperty(SOURCE_TEXT) + .addProperty(COUNT_FLOWFILES) + .build(); + + private static final ConfigurationStep SOURCE_STEP = new ConfigurationStep.Builder() + .name("Source") + .propertyGroups(List.of(SOURCE_GROUP)) + .build(); + + private static final ConnectorPropertyGroup DUPLICATION_GROUP = new ConnectorPropertyGroup.Builder() + .addProperty(NUM_COPIES) + .build(); + + private static final ConfigurationStep DUPLICATION_STEP = new ConfigurationStep.Builder() + .name("Duplication") + .propertyGroups(List.of(DUPLICATION_GROUP)) + .build(); + + private static final ConnectorPropertyGroup DESTINATION_GROUP = new ConnectorPropertyGroup.Builder() + .addProperty(LOG_FLOWFILE_CONTENTS) + .build(); + + private static final ConfigurationStep DESTINATION_STEP = new ConfigurationStep.Builder() + .name("Destination") + .propertyGroups(List.of(DESTINATION_GROUP)) + .build(); + + private static final List configurationSteps = List.of( + SOURCE_STEP, + DUPLICATION_STEP, + DESTINATION_STEP + ); + + @Override + public List getConfigurationSteps() { + return configurationSteps; + } + + @Override + protected void init() { + initialized = true; + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource("flows/generate-duplicate-log-flow.json"); + } + + public boolean isInitialized() { + return initialized; + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + // Apply the updates to the active flow that were made to the working flow + final VersionedExternalFlow versionedFlow = getFlow(workingContext); + getInitializationContext().updateFlow(activeContext, versionedFlow); + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + // Now that configuration is available, update the flow based on configured properties + final VersionedExternalFlow versionedFlow = getFlow(workingContext); + getInitializationContext().updateFlow(workingContext, versionedFlow); + } + + private VersionedExternalFlow getFlow(final FlowContext flowContext) { + final VersionedExternalFlow externalFlow = VersionedFlowUtils.loadFlowFromResource("flows/generate-duplicate-log-flow.json"); + final VersionedProcessGroup versionedProcessGroup = externalFlow.getFlowContents(); + + // Update the flow based on configured properties + updateSourceStep(flowContext, versionedProcessGroup); + updateDuplicationStep(flowContext, versionedProcessGroup); + updateDestinationStep(flowContext, versionedProcessGroup); + + return externalFlow; + } + + private void updateSourceStep(final FlowContext flowContext, final VersionedProcessGroup rootGroup) { + final ConnectorConfigurationContext configContext = flowContext.getConfigurationContext(); + final String sourceText = configContext.getProperty(SOURCE_STEP, SOURCE_TEXT).getValue(); + + final VersionedProcessor sourceTextProcessor = VersionedFlowUtils.findProcessor(rootGroup, + p -> p.getType().equals(OverwriteFlowFile.class.getName())).orElseThrow(); + sourceTextProcessor.getProperties().put(OverwriteFlowFile.CONTENT.getName(), sourceText); + + final boolean count = configContext.getProperty(SOURCE_STEP, COUNT_FLOWFILES).asBoolean(); + if (count) { + final Bundle systemBundle = new Bundle(); + systemBundle.setArtifact("system"); + systemBundle.setGroup("default"); + systemBundle.setVersion("unversioned"); + + final VersionedControllerService controllerService = VersionedFlowUtils.addControllerService(rootGroup, StandardCounterService.class.getName(), systemBundle, "Count"); + rootGroup.getControllerServices().add(controllerService); + + final VersionedProcessor generateProcessor = VersionedFlowUtils.findProcessor(rootGroup, + p -> p.getType().equals(CreateDummyFlowFile.class.getName())).orElseThrow(); + generateProcessor.getProperties().put("Counter Service", controllerService.getIdentifier()); + } + } + + private void updateDuplicationStep(final FlowContext flowContext, final VersionedProcessGroup rootGroup) { + final ConnectorConfigurationContext configContext = flowContext.getConfigurationContext(); + + final int numCopies = configContext.getProperty(DUPLICATION_STEP, NUM_COPIES).asInteger(); + final VersionedProcessor duplicateProcessor = VersionedFlowUtils.findProcessor(rootGroup, + p -> p.getType().equals(DuplicateFlowFile.class.getName())).orElseThrow(); + duplicateProcessor.getProperties().put(DuplicateFlowFile.NUM_DUPLICATES.getName(), String.valueOf(numCopies)); + + // Need to determine how many Connections exist going out of the DuplicateFlowFile processor + // and then add/remove connections as necessary to match the number of copies. + final VersionedProcessGroup duplicatesGroup = VersionedFlowUtils.findGroupForProcessor(rootGroup, duplicateProcessor); + if (duplicatesGroup == null) { + return; + } + + final List outboundConnections = VersionedFlowUtils.findOutboundConnections(rootGroup, duplicateProcessor); + final int currentConnections = outboundConnections.size(); + + if (numCopies > currentConnections) { + // Add new connections for the additional copies + addConnectionsForDuplicates(duplicatesGroup, duplicateProcessor, outboundConnections, numCopies); + } else if (numCopies < currentConnections) { + // Remove excess connections + removeExcessConnections(duplicatesGroup, outboundConnections, numCopies); + } + } + + private void updateDestinationStep(final FlowContext flowContext, final VersionedProcessGroup rootGroup) { + final ConnectorConfigurationContext configContext = flowContext.getConfigurationContext(); + + final boolean logContents = configContext.getProperty(DESTINATION_STEP, LOG_FLOWFILE_CONTENTS).asBoolean(); + if (!logContents) { + return; + } + + final VersionedProcessGroup destinationGroup = rootGroup.getProcessGroups().stream() + .filter(group -> group.getName().equals("Destination")) + .findFirst() + .orElseThrow(); + + // Add a LogFlowFileContents processor to the Destination group. Move the TerminateFlowFile processor down about + // 250 pixels and update connections so that it's Port -> LogFlowFileContents -> TerminateFlowFile + + // Find the existing TerminateFlowFile processor + final VersionedProcessor terminateProcessor = VersionedFlowUtils.findProcessor(destinationGroup, + p -> p.getType().equals("org.apache.nifi.components.connector.processors.TerminateFlowFile")).orElseThrow(); + + // Get the first (and only) input port + final VersionedPort inputPort = destinationGroup.getInputPorts().iterator().next(); + + // Create LogFlowFileContents processor + final VersionedProcessor logProcessor = createLogFlowFileContentsProcessor(destinationGroup, terminateProcessor); + + // Move TerminateFlowFile processor down by 250 pixels + final Position terminatePosition = terminateProcessor.getPosition(); + terminateProcessor.setPosition(new Position(terminatePosition.getX(), terminatePosition.getY() + 250)); + + // Update connections: Port -> LogFlowFileContents -> TerminateFlowFile + updateDestinationConnections(destinationGroup, inputPort, logProcessor, terminateProcessor); + } + + private void addConnectionsForDuplicates(final VersionedProcessGroup duplicatesGroup, final VersionedProcessor duplicateProcessor, + final List existingConnections, final int targetNumCopies) { + + if (existingConnections.isEmpty()) { + return; + } + + // Use the first existing connection as a template for creating new connections + final VersionedConnection templateConnection = existingConnections.getFirst(); + final ConnectableComponent sourceComponent = VersionedFlowUtils.createConnectableComponent(duplicateProcessor); + final ConnectableComponent destinationComponent = templateConnection.getDestination(); + + final int currentConnections = existingConnections.size(); + for (int i = currentConnections + 1; i <= targetNumCopies; i++) { + final Set relationships = Set.of(String.valueOf(i)); + VersionedFlowUtils.addConnection(duplicatesGroup, sourceComponent, destinationComponent, relationships); + } + } + + private void removeExcessConnections(final VersionedProcessGroup duplicatesGroup, final List outboundConnections, final int targetNumCopies) { + final Set connectionsToRemove = new HashSet<>(); + + // Sort connections by relationship name (which should be numeric) and remove the highest numbered ones + final List sortedConnections = new ArrayList<>(outboundConnections); + sortedConnections.sort((c1, c2) -> { + final String rel1 = c1.getSelectedRelationships().iterator().next(); + final String rel2 = c2.getSelectedRelationships().iterator().next(); + try { + final int num1 = Integer.parseInt(rel1); + final int num2 = Integer.parseInt(rel2); + return Integer.compare(num2, num1); // Sort descending to remove highest numbers first + } catch (final NumberFormatException e) { + return rel2.compareTo(rel1); + } + }); + + final int connectionsToRemoveCount = outboundConnections.size() - targetNumCopies; + for (int i = 0; i < connectionsToRemoveCount && i < sortedConnections.size(); i++) { + connectionsToRemove.add(sortedConnections.get(i)); + } + + // Remove the connections from the process group + final Set groupConnections = duplicatesGroup.getConnections(); + if (groupConnections != null) { + groupConnections.removeAll(connectionsToRemove); + } + } + + private void updateDestinationConnections(final VersionedProcessGroup destinationGroup, final VersionedPort inputPort, + final VersionedProcessor logProcessor, final VersionedProcessor terminateProcessor) { + // Find and remove the existing connection from input port to terminate processor + final Set connections = destinationGroup.getConnections(); + if (connections != null) { + connections.stream() + .filter(conn -> conn.getSource().getId().equals(inputPort.getIdentifier())) + .filter(conn -> conn.getDestination().getId().equals(terminateProcessor.getIdentifier())) + .findFirst() + .ifPresent(connections::remove); + } + + // Create connection from input port to LogFlowFileContents processor + final ConnectableComponent inputPortComponent = VersionedFlowUtils.createConnectableComponent(inputPort); + final ConnectableComponent logProcessorComponent = VersionedFlowUtils.createConnectableComponent(logProcessor); + VersionedFlowUtils.addConnection(destinationGroup, inputPortComponent, logProcessorComponent, Set.of("")); + + // Create connection from LogFlowFileContents processor to TerminateFlowFile processor + final ConnectableComponent terminateProcessorComponent = VersionedFlowUtils.createConnectableComponent(terminateProcessor); + VersionedFlowUtils.addConnection(destinationGroup, logProcessorComponent, terminateProcessorComponent, Set.of("success")); + } + + private VersionedProcessor createLogFlowFileContentsProcessor(final VersionedProcessGroup destinationGroup, final VersionedProcessor terminateProcessor) { + // Position new processor where terminate processor is; terminate processor will be moved down later + final Position terminatePosition = terminateProcessor.getPosition(); + final Position logProcessorPosition = new Position(terminatePosition.getX(), terminatePosition.getY()); + + return VersionedFlowUtils.addProcessor( + destinationGroup, + LogFlowFileContents.class.getName(), + terminateProcessor.getBundle(), + "Log FlowFile Contents", + logProcessorPosition + ); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/MissingBundleConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/MissingBundleConnector.java new file mode 100644 index 000000000000..c21b55dfbbaf --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/MissingBundleConnector.java @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** + * A test connector that returns an initial flow containing a processor with a bundle that does not exist. + * This is used to test the behavior when a connector's initial flow references unavailable components. + */ +public class MissingBundleConnector extends AbstractConnector { + + private static final List CONFIGURATION_STEPS = List.of(); + + @Override + public List getConfigurationSteps() { + return CONFIGURATION_STEPS; + } + + @Override + protected void init() { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final VersionedProcessGroup rootGroup = new VersionedProcessGroup(); + rootGroup.setIdentifier(UUID.randomUUID().toString()); + rootGroup.setInstanceIdentifier(UUID.randomUUID().toString()); + rootGroup.setName("Missing Bundle Connector Flow"); + rootGroup.setPosition(new Position(0.0, 0.0)); + rootGroup.setProcessGroups(new HashSet<>()); + rootGroup.setConnections(new HashSet<>()); + rootGroup.setInputPorts(new HashSet<>()); + rootGroup.setOutputPorts(new HashSet<>()); + rootGroup.setControllerServices(new HashSet<>()); + rootGroup.setFunnels(new HashSet<>()); + rootGroup.setLabels(new HashSet<>()); + + final VersionedProcessor missingProcessor = new VersionedProcessor(); + missingProcessor.setIdentifier(UUID.randomUUID().toString()); + missingProcessor.setInstanceIdentifier(UUID.randomUUID().toString()); + missingProcessor.setName("Missing Processor"); + missingProcessor.setType("com.example.nonexistent.MissingProcessor"); + missingProcessor.setPosition(new Position(100.0, 100.0)); + missingProcessor.setScheduledState(ScheduledState.ENABLED); + missingProcessor.setSchedulingPeriod("0 sec"); + missingProcessor.setSchedulingStrategy("TIMER_DRIVEN"); + missingProcessor.setExecutionNode("ALL"); + missingProcessor.setPenaltyDuration("30 sec"); + missingProcessor.setYieldDuration("1 sec"); + missingProcessor.setBulletinLevel("WARN"); + missingProcessor.setRunDurationMillis(0L); + missingProcessor.setConcurrentlySchedulableTaskCount(1); + missingProcessor.setAutoTerminatedRelationships(new HashSet<>()); + missingProcessor.setProperties(Map.of()); + missingProcessor.setPropertyDescriptors(Map.of()); + missingProcessor.setGroupIdentifier(rootGroup.getIdentifier()); + + final Bundle missingBundle = new Bundle(); + missingBundle.setGroup("com.example.nonexistent"); + missingBundle.setArtifact("missing-nar"); + missingBundle.setVersion("1.0.0"); + missingProcessor.setBundle(missingBundle); + + rootGroup.setProcessors(new HashSet<>(List.of(missingProcessor))); + + final VersionedExternalFlow externalFlow = new VersionedExternalFlow(); + externalFlow.setFlowContents(rootGroup); + return externalFlow; + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/OnPropertyModifiedConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/OnPropertyModifiedConnector.java new file mode 100644 index 000000000000..bf5f242a939e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/OnPropertyModifiedConnector.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; +import java.util.Map; + +/** + * A test connector that creates a flow with an OnPropertyModifiedTracker processor. + * The processor's Configured Number property is bound to a parameter, allowing this connector + * to test that onPropertyModified is called when a Connector's applyUpdate changes a parameter value. + */ +public class OnPropertyModifiedConnector extends AbstractConnector { + + private static final String PARAMETER_NAME = "CONFIGURED_NUMBER"; + + static final ConnectorPropertyDescriptor NUMBER_VALUE = new ConnectorPropertyDescriptor.Builder() + .name("Number Value") + .description("The number value to set for the Configured Number parameter") + .type(PropertyType.STRING) + .addValidator(StandardValidators.INTEGER_VALIDATOR) + .required(true) + .build(); + + private static final ConnectorPropertyGroup PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Configuration") + .description("Configuration properties for the OnPropertyModified test") + .addProperty(NUMBER_VALUE) + .build(); + + private static final ConfigurationStep CONFIG_STEP = new ConfigurationStep.Builder() + .name("Configuration") + .description("Configure the number value for testing onPropertyModified") + .propertyGroups(List.of(PROPERTY_GROUP)) + .build(); + + @Override + public VersionedExternalFlow getInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource("flows/on-property-modified-tracker.json"); + } + + @Override + public List getConfigurationSteps() { + return List.of(CONFIG_STEP); + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + final VersionedExternalFlow versionedExternalFlow = getInitialFlow(); + final String number = workingContext.getConfigurationContext().getProperty(CONFIG_STEP, NUMBER_VALUE).getValue(); + VersionedFlowUtils.setParameterValue(versionedExternalFlow, PARAMETER_NAME, number); + getInitializationContext().updateFlow(workingContext, versionedExternalFlow); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final VersionedExternalFlow versionedExternalFlow = getInitialFlow(); + final String number = workingContext.getConfigurationContext().getProperty(CONFIG_STEP, NUMBER_VALUE).getValue(); + VersionedFlowUtils.setParameterValue(versionedExternalFlow, PARAMETER_NAME, number); + getInitializationContext().updateFlow(activeContext, versionedExternalFlow); + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext workingContext) { + return List.of(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/ParameterConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/ParameterConnector.java new file mode 100644 index 000000000000..bbc071aca0eb --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/ParameterConnector.java @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ParameterValue; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; +import java.util.Map; + +public class ParameterConnector extends AbstractConnector { + private volatile boolean initialized = false; + + static final ConnectorPropertyDescriptor TEXT_PROPERTY = new ConnectorPropertyDescriptor.Builder() + .name("Text") + .description("The text to write to FlowFiles") + .type(PropertyType.STRING) + .addValidator(Validator.VALID) + .required(true) + .defaultValue("Hello World") + .build(); + + static final ConnectorPropertyDescriptor SLEEP_DURATION = new ConnectorPropertyDescriptor.Builder() + .name("Sleep Duration") + .description("The duration to sleep when the Sleep Processor is stopped") + .type(PropertyType.STRING) + .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR) + .required(true) + .defaultValue("1 sec") + .build(); + + private static final ConnectorPropertyGroup TEXT_GROUP = new ConnectorPropertyGroup.Builder() + .name("Text Settings") + .description("Settings for the text to write to FlowFiles") + .addProperty(TEXT_PROPERTY) + .addProperty(SLEEP_DURATION) + .build(); + + private static final ConfigurationStep TEXT_STEP = new ConfigurationStep.Builder() + .name("Text Configuration") + .description("Configure the text to be written to FlowFiles") + .propertyGroups(List.of(TEXT_GROUP)) + .build(); + + @Override + protected void init() { + initialized = true; + } + + public boolean isInitialized() { + return initialized; + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return VersionedFlowUtils.loadFlowFromResource("flows/generate-and-log-with-parameter.json"); + } + + @Override + public List getConfigurationSteps() { + return List.of(TEXT_STEP); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) { + try { + updateTextParameter(workingContext, activeContext); + } catch (final FlowUpdateException e) { + getLogger().error("Failed to update parameters", e); + throw new RuntimeException("Failed to update parameters", e); + } + } + + @Override + public void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public void abortUpdate(final FlowContext workingContext, final Throwable throwable) { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext workingContext) { + return List.of(); + } + + private void updateTextParameter(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final ConnectorConfigurationContext configContext = workingContext.getConfigurationContext(); + final String textValue = configContext.getProperty(TEXT_STEP, TEXT_PROPERTY).getValue(); + + // Update the "Text" parameter with the configured property value + final ParameterValue textParameter = new ParameterValue.Builder() + .name("Text") + .value(textValue) + .sensitive(false) + .build(); + + final ParameterValue sleepDurationParameter = new ParameterValue.Builder() + .name("Sleep Duration") + .value(configContext.getProperty(TEXT_STEP, SLEEP_DURATION).getValue()) + .sensitive(false) + .build(); + + final List parameterValues = List.of(textParameter, sleepDurationParameter); + activeContext.getParameterContext().updateParameters(parameterValues); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/SleepingConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/SleepingConnector.java new file mode 100644 index 000000000000..6818b5b059cc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/SleepingConnector.java @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.time.Duration; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +public class SleepingConnector implements Connector { + private final Duration sleepDuration; + + public SleepingConnector(final Duration sleepDuration) { + this.sleepDuration = sleepDuration; + } + + @Override + public void initialize(final ConnectorInitializationContext connectorInitializationContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void start(final FlowContext activeContext) throws FlowUpdateException { + try { + Thread.sleep(sleepDuration); + } catch (final InterruptedException e) { + throw new FlowUpdateException(e); + } + } + + @Override + public void stop(final FlowContext activeContext) throws FlowUpdateException { + try { + Thread.sleep(sleepDuration); + } catch (final InterruptedException e) { + throw new FlowUpdateException(e); + } + } + + @Override + public List validate(final FlowContext flowContext, final ConnectorValidationContext connectorValidationContext) { + return List.of(); + } + + @Override + public List validateConfigurationStep(final ConfigurationStep configurationStep, final ConnectorConfigurationContext connectorConfigurationContext, + final ConnectorValidationContext connectorValidationContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + try { + Thread.sleep(sleepDuration); + } catch (final InterruptedException e) { + throw new FlowUpdateException(e); + } + } + + @Override + public void onConfigurationStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public void abortUpdate(final FlowContext workingContext, final Throwable throwable) { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext activeContext) { + return List.of(); + } + + @Override + public List verify(final FlowContext flowContext) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext, final String filter) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext) { + return List.of(); + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + return CompletableFuture.completedFuture(null); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/StandardConnectorNodeIT.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/StandardConnectorNodeIT.java new file mode 100644 index 000000000000..6b321eca8e86 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/StandardConnectorNodeIT.java @@ -0,0 +1,675 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.connector.processors.CreateDummyFlowFile; +import org.apache.nifi.components.connector.processors.DuplicateFlowFile; +import org.apache.nifi.components.connector.processors.LogFlowFileContents; +import org.apache.nifi.components.connector.processors.OnPropertyModifiedTracker; +import org.apache.nifi.components.connector.processors.OverwriteFlowFile; +import org.apache.nifi.components.connector.processors.TerminateFlowFile; +import org.apache.nifi.components.connector.secrets.ParameterProviderSecretsManager; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.components.connector.services.CounterService; +import org.apache.nifi.components.state.StateManagerProvider; +import org.apache.nifi.components.validation.StandardVerifiableComponentFactory; +import org.apache.nifi.components.validation.ValidationState; +import org.apache.nifi.components.validation.ValidationStatus; +import org.apache.nifi.components.validation.ValidationTrigger; +import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.StandardConnection; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.GarbageCollectionLog; +import org.apache.nifi.controller.MockStateManagerProvider; +import org.apache.nifi.controller.NodeTypeProvider; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.ReloadComponent; +import org.apache.nifi.controller.flow.StandardFlowManager; +import org.apache.nifi.controller.flowanalysis.FlowAnalyzer; +import org.apache.nifi.controller.queue.DropFlowFileRequest; +import org.apache.nifi.controller.queue.DropFlowFileState; +import org.apache.nifi.controller.queue.FlowFileQueue; +import org.apache.nifi.controller.queue.FlowFileQueueFactory; +import org.apache.nifi.controller.queue.LoadBalanceCompression; +import org.apache.nifi.controller.queue.LoadBalanceStrategy; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.repository.ContentRepository; +import org.apache.nifi.controller.repository.FlowFileEventRepository; +import org.apache.nifi.controller.repository.FlowFileRecord; +import org.apache.nifi.controller.repository.FlowFileRepository; +import org.apache.nifi.controller.scheduling.LifecycleStateManager; +import org.apache.nifi.controller.scheduling.RepositoryContextFactory; +import org.apache.nifi.controller.scheduling.SchedulingAgent; +import org.apache.nifi.controller.scheduling.StandardLifecycleStateManager; +import org.apache.nifi.controller.scheduling.StandardProcessScheduler; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.controller.service.ControllerServiceProvider; +import org.apache.nifi.engine.FlowEngine; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.mock.MockNodeTypeProvider; +import org.apache.nifi.nar.ExtensionDiscoveringManager; +import org.apache.nifi.nar.StandardExtensionDiscoveringManager; +import org.apache.nifi.nar.SystemBundle; +import org.apache.nifi.parameter.Parameter; +import org.apache.nifi.parameter.ParameterContext; +import org.apache.nifi.parameter.ParameterContextManager; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.provenance.ProvenanceRepository; +import org.apache.nifi.python.PythonBridge; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.scheduling.SchedulingStrategy; +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.validation.RuleViolationsManager; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.function.Predicate; + +import static java.util.Objects.requireNonNull; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyCollection; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.nullable; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class StandardConnectorNodeIT { + + private StandardProcessScheduler processScheduler; + private StandardFlowManager flowManager; + private FlowEngine componentLifecycleThreadPool; + private ConnectorRepository connectorRepository; + + @BeforeEach + public void setup() { + final ControllerServiceProvider controllerServiceProvider = mock(ControllerServiceProvider.class); + when(controllerServiceProvider.disableControllerServicesAsync(anyCollection())).thenReturn(CompletableFuture.completedFuture(null)); + connectorRepository = new StandardConnectorRepository(); + + final SecretsManager secretsManager = new ParameterProviderSecretsManager(); + final ConnectorRepositoryInitializationContext repoInitContext = mock(ConnectorRepositoryInitializationContext.class); + when(repoInitContext.getSecretsManager()).thenReturn(secretsManager); + connectorRepository.initialize(repoInitContext); + + final ExtensionDiscoveringManager extensionManager = new StandardExtensionDiscoveringManager(); + final BulletinRepository bulletinRepository = mock(BulletinRepository.class); + final StateManagerProvider stateManagerProvider = new MockStateManagerProvider(); + final LifecycleStateManager lifecycleStateManager = new StandardLifecycleStateManager(); + final ReloadComponent reloadComponent = mock(ReloadComponent.class); + + final NiFiProperties nifiProperties = NiFiProperties.createBasicNiFiProperties("src/test/resources/conf/nifi.properties"); + + final FlowController flowController = mock(FlowController.class); + when(flowController.isInitialized()).thenReturn(true); + when(flowController.getExtensionManager()).thenReturn(extensionManager); + when(flowController.getStateManagerProvider()).thenReturn(stateManagerProvider); + when(flowController.getReloadComponent()).thenReturn(reloadComponent); + when(flowController.getVerifiableComponentFactory()).thenReturn(new StandardVerifiableComponentFactory(flowController, nifiProperties)); + + final RepositoryContextFactory repoContextFactory = mock(RepositoryContextFactory.class); + final FlowFileRepository flowFileRepo = mock(FlowFileRepository.class); + final ProvenanceRepository provRepo = mock(ProvenanceRepository.class); + final ContentRepository contentRepo = mock(ContentRepository.class); + when(repoContextFactory.getFlowFileRepository()).thenReturn(flowFileRepo); + when(repoContextFactory.getProvenanceRepository()).thenReturn(provRepo); + when(repoContextFactory.getContentRepository()).thenReturn(contentRepo); + + when(flowController.getRepositoryContextFactory()).thenReturn(repoContextFactory); + when(flowController.getGarbageCollectionLog()).thenReturn(mock(GarbageCollectionLog.class)); + when(flowController.getControllerServiceProvider()).thenReturn(controllerServiceProvider); + when(flowController.getProvenanceRepository()).thenReturn(provRepo); + when(flowController.getBulletinRepository()).thenReturn(bulletinRepository); + when(flowController.getLifecycleStateManager()).thenReturn(lifecycleStateManager); + when(flowController.getFlowFileEventRepository()).thenReturn(mock(FlowFileEventRepository.class)); + when(flowController.getConnectorRepository()).thenReturn(connectorRepository); + when(flowController.getValidationTrigger()).thenReturn(mock(ValidationTrigger.class)); + when(flowController.getConnectorValidationTrigger()).thenReturn(mock(ConnectorValidationTrigger.class)); + + doAnswer(invocation -> { + return createConnection(invocation.getArgument(0), invocation.getArgument(1), invocation.getArgument(2), invocation.getArgument(3), invocation.getArgument(4)); + }).when(flowController).createConnection(anyString(), nullable(String.class), any(Connectable.class), any(Connectable.class), anyCollection()); + + final FlowFileEventRepository flowFileEventRepository = mock(FlowFileEventRepository.class); + final ParameterContextManager parameterContextManager = mock(ParameterContextManager.class); + + final NodeTypeProvider nodeTypeProvider = new MockNodeTypeProvider(); + componentLifecycleThreadPool = new FlowEngine(4, "Component Lifecycle Thread Pool", true); + processScheduler = new StandardProcessScheduler(componentLifecycleThreadPool, extensionManager, nodeTypeProvider, () -> controllerServiceProvider, + reloadComponent, stateManagerProvider, nifiProperties, lifecycleStateManager); + when(flowController.getProcessScheduler()).thenReturn(processScheduler); + processScheduler.setSchedulingAgent(SchedulingStrategy.TIMER_DRIVEN, mock(SchedulingAgent.class)); + processScheduler.setSchedulingAgent(SchedulingStrategy.CRON_DRIVEN, mock(SchedulingAgent.class)); + + final Bundle systemBundle = SystemBundle.create(nifiProperties); + extensionManager.discoverExtensions(systemBundle, Set.of()); + + flowManager = new StandardFlowManager(nifiProperties, null, flowController, flowFileEventRepository, parameterContextManager); + flowManager.initialize(controllerServiceProvider, mock(PythonBridge.class), mock(FlowAnalyzer.class), mock(RuleViolationsManager.class)); + final ProcessGroup rootGroup = flowManager.createProcessGroup("root"); + rootGroup.setName("Root"); + flowManager.setRootGroup(rootGroup); + + when(flowController.getFlowManager()).thenReturn(flowManager); + } + + @AfterEach + public void tearDown() { + if (componentLifecycleThreadPool != null) { + componentLifecycleThreadPool.shutdown(); + } + } + + @Test + public void testConnectorDynamicallyCreatingFlow() throws FlowUpdateException { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + + // Configure the connector to log FlowFile contents, make 5 copies, and set text to "Second Iteration." + final ConnectorConfiguration configuration = createConnectorConfiguration("Second Iteration.", 5, true, false); + configure(connectorNode, configuration); + + final List processorsAfterUpdate = rootGroup.findAllProcessors(); + assertEquals(5, processorsAfterUpdate.size()); + + final List processorTypesAfterUpdate = processorsAfterUpdate.stream().map(ProcessorNode::getCanonicalClassName).toList(); + assertTrue(processorTypesAfterUpdate.contains(CreateDummyFlowFile.class.getName())); + assertTrue(processorTypesAfterUpdate.contains(OverwriteFlowFile.class.getName())); + assertTrue(processorTypesAfterUpdate.contains(TerminateFlowFile.class.getName())); + assertTrue(processorTypesAfterUpdate.contains(DuplicateFlowFile.class.getName())); + assertTrue(processorTypesAfterUpdate.contains(LogFlowFileContents.class.getName())); + + // Verify the OverwriteFlowFile processor has the correct content configured + final ProcessorNode overwriteProcessor = processorsAfterUpdate.stream() + .filter(p -> p.getCanonicalClassName().equals(OverwriteFlowFile.class.getName())) + .findFirst() + .orElseThrow(); + assertEquals("Second Iteration.", overwriteProcessor.getEffectivePropertyValue(OverwriteFlowFile.CONTENT)); + + // Verify the DuplicateFlowFile processor is configured for 5 copies + final ProcessorNode duplicateProcessor = processorsAfterUpdate.stream() + .filter(p -> p.getCanonicalClassName().equals(DuplicateFlowFile.class.getName())) + .findFirst() + .orElseThrow(); + assertEquals("5", duplicateProcessor.getEffectivePropertyValue(DuplicateFlowFile.NUM_DUPLICATES)); + + // Verify that all of the Connections were created + assertEquals(5, duplicateProcessor.getConnections().size()); + final String outputPortId = duplicateProcessor.getProcessGroup().getOutputPorts().iterator().next().getIdentifier(); + for (final Connection connection : duplicateProcessor.getConnections()) { + assertEquals(duplicateProcessor.getIdentifier(), connection.getSource().getIdentifier()); + assertEquals(outputPortId, connection.getDestination().getIdentifier()); + } + } + + private void configure(final ConnectorNode connectorNode, final ConnectorConfiguration configuration) throws FlowUpdateException { + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + for (final NamedStepConfiguration stepConfig : configuration.getNamedStepConfigurations()) { + connectorNode.setConfiguration(stepConfig.stepName(), stepConfig.configuration()); + } + connectorNode.applyUpdate(); + } + + private ConnectorNode initializeDynamicFlowConnector() { + final ConnectorNode connectorNode = flowManager.createConnector(DynamicFlowConnector.class.getName(), "dynamic-flow-connector", SystemBundle.SYSTEM_BUNDLE_COORDINATE, true, true); + assertNotNull(connectorNode); + + final Connector connector = connectorNode.getConnector(); + assertNotNull(connector); + assertInstanceOf(DynamicFlowConnector.class, connector); + + final DynamicFlowConnector flowConnector = (DynamicFlowConnector) connector; + assertTrue(flowConnector.isInitialized()); + + assertEquals(List.of(connectorNode), connectorRepository.getConnectors()); + + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + assertEquals(3, rootGroup.getProcessGroups().size()); + + final List initialProcessors = rootGroup.findAllProcessors(); + assertEquals(4, initialProcessors.size()); + final List initialProcessorTypes = initialProcessors.stream().map(ProcessorNode::getComponentType).toList(); + assertTrue(initialProcessorTypes.contains(CreateDummyFlowFile.class.getSimpleName())); + assertTrue(initialProcessorTypes.contains(OverwriteFlowFile.class.getSimpleName())); + assertTrue(initialProcessorTypes.contains(TerminateFlowFile.class.getSimpleName())); + assertTrue(initialProcessorTypes.contains(DuplicateFlowFile.class.getSimpleName())); + + return connectorNode; + } + + private ConnectorNode initializeParameterConnector() { + final ConnectorNode connectorNode = flowManager.createConnector(ParameterConnector.class.getName(), "parameter-connector", SystemBundle.SYSTEM_BUNDLE_COORDINATE, true, true); + assertNotNull(connectorNode); + assertEquals(List.of(connectorNode), connectorRepository.getConnectors()); + + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + assertEquals(3, rootGroup.getProcessors().size()); + + return connectorNode; + } + + // Test a scenario where the connector is initialized and the Connector then changes the value of a parameter. + // This should result in any Processor that references that parameter being stopped and restarted. It should not result + // in non-referencing Processors being restarted. + @Test + public void testParameterUpdateRestartsReferencingProcessors() throws FlowUpdateException { + final ConnectorNode connectorNode = initializeParameterConnector(); + final Connector connector = connectorNode.getConnector(); + assertNotNull(connector); + assertInstanceOf(ParameterConnector.class, connector); + + final ParameterConnector flowConnector = (ParameterConnector) connector; + assertTrue(flowConnector.isInitialized()); + + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ParameterContext parameterContext = rootGroup.getParameterContext(); + assertNotNull(parameterContext); + assertEquals(2, parameterContext.getParameters().size()); + final Optional optionalParameter = parameterContext.getParameter("Text"); + assertTrue(optionalParameter.isPresent()); + + final Parameter textParameter = optionalParameter.get(); + assertEquals("Hello", textParameter.getValue()); + + // Set the value of the 'Text' property to Hi. This should result in the parameter context being updated. + final StepConfiguration sourceStepConfig = new StepConfiguration(Map.of("Text", new StringLiteralValue("Hi."))); + final NamedStepConfiguration sourceConfigurationStepConfiguration = new NamedStepConfiguration("Text Configuration", sourceStepConfig); + final ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration(Set.of(sourceConfigurationStepConfiguration)); + configure(connectorNode, connectorConfiguration); + + assertEquals("Hi.", parameterContext.getParameter("Text").orElseThrow().getValue()); + } + + @Test + @Timeout(10) + public void testOnPropertyModifiedCalledOnApplyUpdate() throws FlowUpdateException { + final ConnectorNode connectorNode = flowManager.createConnector(OnPropertyModifiedConnector.class.getName(), + "on-property-modified-connector", SystemBundle.SYSTEM_BUNDLE_COORDINATE, true, true); + assertNotNull(connectorNode); + + final StepConfiguration initialConfig = new StepConfiguration(Map.of("Number Value", new StringLiteralValue("0"))); + final NamedStepConfiguration initialStepConfig = new NamedStepConfiguration("Configuration", initialConfig); + configure(connectorNode, new ConnectorConfiguration(Set.of(initialStepConfig))); + + final ProcessGroup activeGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ProcessorNode processorNode = activeGroup.getProcessors().iterator().next(); + final OnPropertyModifiedTracker tracker = (OnPropertyModifiedTracker) processorNode.getProcessor(); + + assertEquals(0, tracker.getPropertyChangeCount()); + + final StepConfiguration updatedConfig = new StepConfiguration(Map.of("Number Value", new StringLiteralValue("1"))); + final NamedStepConfiguration updatedStepConfig = new NamedStepConfiguration("Configuration", updatedConfig); + + connectorNode.setConfiguration("Configuration", updatedStepConfig.configuration()); + + final ProcessGroup workingGroup = connectorNode.getWorkingFlowContext().getManagedProcessGroup(); + final ProcessorNode workingProcessorNode = workingGroup.getProcessors().iterator().next(); + final OnPropertyModifiedTracker workingTracker = (OnPropertyModifiedTracker) workingProcessorNode.getProcessor(); + + assertEquals(1, workingTracker.getPropertyChangeCount()); + assertEquals("0", workingTracker.getPropertyChanges().getFirst().oldValue()); + assertEquals("1", workingTracker.getPropertyChanges().getFirst().newValue()); + + workingTracker.clearPropertyChanges(); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.applyUpdate(); + + assertEquals(1, tracker.getPropertyChangeCount()); + assertEquals("0", tracker.getPropertyChanges().getFirst().oldValue()); + assertEquals("1", tracker.getPropertyChanges().getFirst().newValue()); + + // Ensure that no parameter contexts are registered + final Set registeredContexts = flowManager.getParameterContextManager().getParameterContexts(); + assertEquals(Set.of(), registeredContexts); + } + + @Test + public void testControllerServices() throws FlowUpdateException { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + + final ConnectorConfiguration configuration = createConnectorConfiguration("Second Iteration", 5, true, true); + configure(connectorNode, configuration); + + final ProcessGroup managedGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final Set serviceNodes = managedGroup.getControllerServices(true); + assertNotNull(serviceNodes); + assertEquals(1, serviceNodes.size()); + assertInstanceOf(CounterService.class, serviceNodes.iterator().next().getControllerServiceImplementation()); + } + + @Test + public void testUpdateProcessorPropertyDataQueued() throws FlowUpdateException { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final Connection connection = queueDataBySource(rootGroup, "Create FlowFile"); + + // Update Connector Property to ensure that the change is allowed with data queued + final ConnectorConfiguration configuration = createConnectorConfiguration("Second Iteration", 5, true, false); + configure(connectorNode, configuration); + + assertEquals(1, connection.getFlowFileQueue().size().getObjectCount()); + } + + @Test + public void testRemoveConnectionDataQueued() throws FlowUpdateException { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + + // Create a configuration that will result in the LogFlowFileContents processor being removed. + // Because the component is being removed, it should drain the queues before doing so. + final ConnectorConfiguration addLogConfiguration = createConnectorConfiguration("Second Iteration", 5, true, false); + configure(connectorNode, addLogConfiguration); + + // Queue data between LogFlowFileContents and TerminateFlowFile + final Connection connection = queueDataByDestination(rootGroup, "Terminate FlowFile"); + + // Create a configuration that will result in the LogFlowFileContents processor being removed. + // Because the component is being removed and there's data queued in its incoming connection, it should fail. + final ConnectorConfiguration removeLogConfiguration = createConnectorConfiguration("Second Iteration", 5, false, false); + + final Throwable cause = assertThrows(FlowUpdateException.class, () -> configure(connectorNode, removeLogConfiguration)); + connectorNode.abortUpdate(cause); + + rootGroup.findAllConnections().contains(connection); + assertFalse(connection.getFlowFileQueue().isEmpty()); + } + + + @Test + public void testDynamicProperties() throws IOException, FlowUpdateException { + final ConnectorNode connectorNode = flowManager.createConnector(DynamicAllowableValuesConnector.class.getName(), "dynamic-allowable-values-connector", + SystemBundle.SYSTEM_BUNDLE_COORDINATE, true, true); + assertNotNull(connectorNode); + + assertEquals(List.of("File", "Colors"), getConfigurationStepNames(connectorNode)); + + final Path tempFile = Files.createTempFile("StandardConnectorNodeIT", ".txt"); + Files.writeString(tempFile, String.join("\n", "red", "blue", "yellow")); + + final ConnectorConfiguration configuration = createFileConfiguration(tempFile.toFile().getAbsolutePath()); + configure(connectorNode, configuration); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + assertEquals(List.of("File", "Colors"), getConfigurationStepNames(connectorNode)); + + final List allowableValues = connectorNode.fetchAllowableValues("Colors", "First Primary Color").stream() + .map(DescribedValue::getValue) + .toList(); + assertEquals(List.of("red", "blue", "yellow"), allowableValues); + } + + @Test + public void testSimpleValidation() throws FlowUpdateException { + final ConnectorNode connectorNode = flowManager.createConnector(DynamicAllowableValuesConnector.class.getName(), "dynamic-allowable-values-connector", + SystemBundle.SYSTEM_BUNDLE_COORDINATE, true, true); + assertNotNull(connectorNode); + + assertEquals(List.of("File", "Colors"), getConfigurationStepNames(connectorNode)); + + final ConnectorConfiguration configuration = createFileConfiguration("/non/existent/file"); + configure(connectorNode, configuration); + + final ValidationState validationState = connectorNode.performValidation(); + assertNotNull(validationState); + assertEquals(ValidationStatus.INVALID, validationState.getStatus()); + assertEquals(2, validationState.getValidationErrors().size()); + + final boolean hasFileError = validationState.getValidationErrors().stream() + .anyMatch(result -> result.getInput() != null && result.getInput().contains("/non/existent/file")); + assertTrue(hasFileError); + + final boolean hasColorError = validationState.getValidationErrors().stream() + .anyMatch(result -> result.getSubject() != null && result.getSubject().contains("First Primary Color")); + assertTrue(hasColorError); + + final File colorsFile = new File("src/test/resources/colors.txt"); + final ConnectorConfiguration validConfig = createFileAndColorsConfiguration(colorsFile.getAbsolutePath(), "red"); + configure(connectorNode, validConfig); + + final ValidationState updatedValidationState = connectorNode.performValidation(); + assertEquals(ValidationStatus.VALID, updatedValidationState.getStatus(), + "Expected valid state but invalid due to " + updatedValidationState.getValidationErrors()); + assertEquals(List.of(), updatedValidationState.getValidationErrors()); + } + + @Test + public void testValidationWithParameterContext() throws FlowUpdateException { + final ConnectorNode connectorNode = initializeParameterConnector(); + + final ValidationState initialValidationState = connectorNode.performValidation(); + assertNotNull(initialValidationState); + assertEquals(ValidationStatus.VALID, initialValidationState.getStatus()); + assertEquals(List.of(), initialValidationState.getValidationErrors()); + + final StepConfiguration sourceStepConfig = new StepConfiguration(Map.of("Sleep Duration", new StringLiteralValue("Hi."))); + final NamedStepConfiguration sourceConfigurationStepConfiguration = new NamedStepConfiguration("Text Configuration", sourceStepConfig); + final ConnectorConfiguration connectorConfiguration = new ConnectorConfiguration(Set.of(sourceConfigurationStepConfiguration)); + configure(connectorNode, connectorConfiguration); + + final ValidationState validationState = connectorNode.performValidation(); + assertNotNull(validationState); + assertEquals(ValidationStatus.INVALID, validationState.getStatus()); + assertEquals(1, validationState.getValidationErrors().size()); + } + + @Test + public void testPurgeFlowFilesEmptiesQueues() throws ExecutionException, InterruptedException, TimeoutException { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + + final Connection connection = queueDataBySource(rootGroup, "Create FlowFile"); + assertEquals(1, connection.getFlowFileQueue().size().getObjectCount()); + + final Future purgeFuture = connectorNode.purgeFlowFiles("test-user"); + purgeFuture.get(10, TimeUnit.SECONDS); + + assertTrue(connection.getFlowFileQueue().isEmpty()); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + } + + @Test + public void testPurgeFlowFilesMultipleQueues() throws ExecutionException, InterruptedException, TimeoutException { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + + final Connection connection1 = queueDataBySource(rootGroup, "Create FlowFile"); + final Connection connection2 = queueDataByDestination(rootGroup, "Terminate FlowFile"); + assertEquals(1, connection1.getFlowFileQueue().size().getObjectCount()); + assertEquals(1, connection2.getFlowFileQueue().size().getObjectCount()); + + final Future purgeFuture = connectorNode.purgeFlowFiles("test-user"); + purgeFuture.get(10, TimeUnit.SECONDS); + + assertTrue(connection1.getFlowFileQueue().isEmpty()); + assertTrue(connection2.getFlowFileQueue().isEmpty()); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + } + + @Test + public void testPurgeFlowFilesRequiresStoppedState() { + final ConnectorNode connectorNode = initializeDynamicFlowConnector(); + final ProcessGroup rootGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + queueDataBySource(rootGroup, "Create FlowFile"); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.start(componentLifecycleThreadPool); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + + final IllegalStateException exception = assertThrows(IllegalStateException.class, () -> connectorNode.purgeFlowFiles("test-user")); + assertTrue(exception.getMessage().contains("must be STOPPED")); + + connectorNode.stop(componentLifecycleThreadPool); + } + + @Test + public void testInitialFlowWithMissingBundleResultsInInvalidConnector() { + final ConnectorNode connectorNode = flowManager.createConnector(MissingBundleConnector.class.getName(), "missing-bundle-connector", SystemBundle.SYSTEM_BUNDLE_COORDINATE, true, true); + assertNotNull(connectorNode); + + assertFalse(connectorNode.isExtensionMissing()); + + final ValidationState validationState = connectorNode.performValidation(); + assertNotNull(validationState); + assertEquals(ValidationStatus.INVALID, validationState.getStatus()); + assertEquals(1, validationState.getValidationErrors().size()); + + final String explanation = validationState.getValidationErrors().iterator().next().getExplanation(); + assertTrue(explanation.contains("com.example.nonexistent:missing-nar:1.0.0"), "Expected explanation to mention the missing bundle coordinates but was: " + explanation); + assertTrue(explanation.contains("com.example.nonexistent.MissingProcessor"), "Expected explanation to mention the missing processor type but was: " + explanation); + } + + private List getConfigurationStepNames(final ConnectorNode connectorNode) { + return connectorNode.getConfigurationSteps().stream() + .map(ConfigurationStep::getName) + .toList(); + } + + private Connection queueDataBySource(final ProcessGroup group, final String sourceComponentName) { + return queueData(group, conn -> conn.getSource().getName().equals(sourceComponentName)); + } + + private Connection queueDataByDestination(final ProcessGroup group, final String destinationComponentName) { + return queueData(group, conn -> conn.getDestination().getName().equals(destinationComponentName)); + } + + private Connection queueData(final ProcessGroup group, final Predicate connectionTest) { + final Connection connection = group.findAllConnections().stream() + .filter(connectionTest) + .findFirst() + .orElseThrow(); + + final FlowFileRecord flowFile = mock(FlowFileRecord.class); + connection.getFlowFileQueue().put(flowFile); + return connection; + } + + private Connection createConnection(final String id, final String name, final Connectable source, final Connectable destination, final Collection relationshipNames) { + final List relationships = relationshipNames.stream() + .map(relName -> new Relationship.Builder().name(relName).build()) + .toList(); + + final FlowFileQueue flowFileQueue = mock(FlowFileQueue.class); + final List flowFileList = new ArrayList<>(); + + // Update mock to add FlowFiles to the queue + doAnswer(invocation -> { + flowFileList.add(invocation.getArgument(0)); + return null; + }).when(flowFileQueue).put(any(FlowFileRecord.class)); + + // Update mock to return queue size and isEmpty status + when(flowFileQueue.size()).thenAnswer(invocation -> new QueueSize(flowFileList.size(), flowFileList.size())); + when(flowFileQueue.isEmpty()).thenAnswer(invocation -> flowFileList.isEmpty()); + when(flowFileQueue.getLoadBalanceStrategy()).thenReturn(LoadBalanceStrategy.DO_NOT_LOAD_BALANCE); + when(flowFileQueue.getLoadBalanceCompression()).thenReturn(LoadBalanceCompression.DO_NOT_COMPRESS); + + // Mock dropFlowFiles to clear the list and return a completed status + when(flowFileQueue.dropFlowFiles(anyString(), anyString())).thenAnswer(invocation -> { + final String requestId = invocation.getArgument(0); + final int originalCount = flowFileList.size(); + flowFileList.clear(); + + final DropFlowFileRequest dropRequest = new DropFlowFileRequest(requestId); + dropRequest.setOriginalSize(new QueueSize(originalCount, originalCount)); + dropRequest.setCurrentSize(new QueueSize(0, 0)); + dropRequest.setDroppedSize(new QueueSize(originalCount, originalCount)); + dropRequest.setState(DropFlowFileState.COMPLETE); + return dropRequest; + }); + + final FlowFileQueueFactory flowFileQueueFactory = (loadBalanceStrategy, partitioningAttribute, processGroup) -> flowFileQueue; + + final Connection connection = new StandardConnection.Builder(processScheduler) + .id(id) + .name(name) + .processGroup(destination.getProcessGroup()) + .relationships(relationships) + .source(requireNonNull(source)) + .destination(destination) + .flowFileQueueFactory(flowFileQueueFactory) + .build(); + + return connection; + } + + private ConnectorConfiguration createConnectorConfiguration(final String sourceText, final int numberOfCopies, final boolean logContents, final boolean countFlowFiles) { + // Source configuration step + final StepConfiguration sourceStepConfig = new StepConfiguration(Map.of( + "Source Text", new StringLiteralValue(sourceText), + "Count FlowFiles", new StringLiteralValue(Boolean.toString(countFlowFiles)))); + final NamedStepConfiguration sourceConfigurationStepConfiguration = new NamedStepConfiguration("Source", sourceStepConfig); + + // Duplication configuration step + final StepConfiguration duplicationStepConfig = new StepConfiguration(Map.of("Number of Copies", new StringLiteralValue(Integer.toString(numberOfCopies)))); + final NamedStepConfiguration duplicationConfigurationStepConfiguration = new NamedStepConfiguration("Duplication", duplicationStepConfig); + + // Destination configuration step + final StepConfiguration destinationStepConfig = new StepConfiguration(Map.of("Log FlowFile Contents", new StringLiteralValue(Boolean.toString(logContents)))); + final NamedStepConfiguration destinationConfigurationStepConfiguration = new NamedStepConfiguration("Destination", destinationStepConfig); + + return new ConnectorConfiguration(Set.of(sourceConfigurationStepConfiguration, duplicationConfigurationStepConfiguration, destinationConfigurationStepConfiguration)); + } + + private ConnectorConfiguration createFileConfiguration(final String filename) { + final StepConfiguration fileStepConfig = new StepConfiguration(Map.of("File Path", new StringLiteralValue(filename))); + final NamedStepConfiguration fileConfigurationStepConfiguration = new NamedStepConfiguration("File", fileStepConfig); + + return new ConnectorConfiguration(Set.of(fileConfigurationStepConfiguration)); + } + + private ConnectorConfiguration createFileAndColorsConfiguration(final String filename, final String color) { + final StepConfiguration fileStepConfig = new StepConfiguration(Map.of("File Path", new StringLiteralValue(filename))); + final NamedStepConfiguration fileConfigurationStepConfiguration = new NamedStepConfiguration("File", fileStepConfig); + + final StepConfiguration colorsStepConfig = new StepConfiguration(Map.of("First Primary Color", new StringLiteralValue(color))); + final NamedStepConfiguration colorsConfigurationStepConfiguration = new NamedStepConfiguration("Colors", colorsStepConfig); + + return new ConnectorConfiguration(Set.of(fileConfigurationStepConfiguration, colorsConfigurationStepConfiguration)); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardComponentBundleLookup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardComponentBundleLookup.java new file mode 100644 index 000000000000..c0916d433593 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardComponentBundleLookup.java @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.bundle.BundleDetails; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.nar.ExtensionManager; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.util.List; +import java.util.Optional; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestStandardComponentBundleLookup { + private static final String COMPONENT_TYPE = "org.apache.nifi.processors.TestProcessor"; + + private ExtensionManager extensionManager; + private StandardComponentBundleLookup lookup; + + @BeforeEach + void setup() { + extensionManager = mock(ExtensionManager.class); + lookup = new StandardComponentBundleLookup(extensionManager); + } + + @Nested + class GetLatestBundle { + @Test + void testMinorVersion() { + setupBundles( + createBundle("group", "artifact", "1.0.0"), + createBundle("group", "artifact", "1.2.0") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("1.2.0", latestBundle.get().getVersion()); + } + + @Test + void testMajorVersion() { + setupBundles( + createBundle("group", "artifact", "1.0.0"), + createBundle("group", "artifact", "2.0.0") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("2.0.0", latestBundle.get().getVersion()); + } + + @Test + void testMoreDigitsLater() { + setupBundles( + createBundle("group", "artifact", "1.0"), + createBundle("group", "artifact", "1.0.1") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("1.0.1", latestBundle.get().getVersion()); + } + + @Test + void testMoreDigitsFirst() { + setupBundles( + createBundle("group", "artifact", "1.0.1"), + createBundle("group", "artifact", "1.0") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("1.0.1", latestBundle.get().getVersion()); + } + + @Test + void testWithSnapshotAndSameVersion() { + setupBundles( + createBundle("group", "artifact", "1.0.0-SNAPSHOT"), + createBundle("group", "artifact", "1.0.0") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("1.0.0", latestBundle.get().getVersion()); + } + + @Test + void testWithSnapshotAndDifferentVersion() { + setupBundles( + createBundle("group", "artifact", "1.0.1-SNAPSHOT"), + createBundle("group", "artifact", "1.0.0") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("1.0.1-SNAPSHOT", latestBundle.get().getVersion()); + } + + @Test + void testEmptyList() { + setupBundles(); + assertTrue(lookup.getLatestBundle(COMPONENT_TYPE).isEmpty()); + } + + @Test + void testNonNumericVersionPart() { + setupBundles( + createBundle("group", "artifact", "4.0.0"), + createBundle("group", "artifact", "4.0.next") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("4.0.0", latestBundle.get().getVersion()); + } + + @Test + void testFullyNonNumericVersionVsNumeric() { + setupBundles( + createBundle("group", "artifact", "1.0.0"), + createBundle("group", "artifact", "undefined") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("1.0.0", latestBundle.get().getVersion()); + } + + @Test + void testTwoFullyNonNumericVersions() { + setupBundles( + createBundle("group", "artifact", "undefined"), + createBundle("group", "artifact", "unknown") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("unknown", latestBundle.get().getVersion()); + } + + @Test + void testQualifierOrdering() { + setupBundles( + createBundle("group", "artifact", "2.0.0-SNAPSHOT"), + createBundle("group", "artifact", "2.0.0-M1"), + createBundle("group", "artifact", "2.0.0-M4"), + createBundle("group", "artifact", "2.0.0-RC1"), + createBundle("group", "artifact", "2.0.0-RC2"), + createBundle("group", "artifact", "2.0.0") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("2.0.0", latestBundle.get().getVersion()); + } + + @Test + void testQualifierOrderingWithoutRelease() { + setupBundles( + createBundle("group", "artifact", "2.0.0-SNAPSHOT"), + createBundle("group", "artifact", "2.0.0-M1"), + createBundle("group", "artifact", "2.0.0-M4"), + createBundle("group", "artifact", "2.0.0-RC1"), + createBundle("group", "artifact", "2.0.0-RC2") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("2.0.0-RC2", latestBundle.get().getVersion()); + } + + @Test + void testMilestoneVersionsOrdering() { + setupBundles( + createBundle("group", "artifact", "2.0.0-M1"), + createBundle("group", "artifact", "2.0.0-M4"), + createBundle("group", "artifact", "2.0.0-SNAPSHOT") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("2.0.0-M4", latestBundle.get().getVersion()); + } + + @Test + void testCalendarDateFormat() { + setupBundles( + createBundle("group", "artifact", "2025.12.31"), + createBundle("group", "artifact", "2026.01.01") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("2026.01.01", latestBundle.get().getVersion()); + } + + @Test + void testCalendarDateFormatWithBuildNumber() { + setupBundles( + createBundle("group", "artifact", "2025.12.31.999"), + createBundle("group", "artifact", "2026.01.01.451") + ); + + final Optional latestBundle = lookup.getLatestBundle(COMPONENT_TYPE); + assertTrue(latestBundle.isPresent()); + assertEquals("2026.01.01.451", latestBundle.get().getVersion()); + } + + private void setupBundles(final org.apache.nifi.bundle.Bundle... bundles) { + when(extensionManager.getBundles(COMPONENT_TYPE)).thenReturn(List.of(bundles)); + } + } + + private org.apache.nifi.bundle.Bundle createBundle(final String group, final String artifact, final String version) { + final BundleCoordinate coordinate = new BundleCoordinate(group, artifact, version); + final BundleDetails details = new BundleDetails.Builder() + .coordinate(coordinate) + .workingDir(new File(".")) + .build(); + return new org.apache.nifi.bundle.Bundle(details, this.getClass().getClassLoader()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorConfigurationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorConfigurationContext.java new file mode 100644 index 000000000000..6a73137ac762 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorConfigurationContext.java @@ -0,0 +1,256 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.secrets.SecretProvider; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestStandardConnectorConfigurationContext { + private StandardConnectorConfigurationContext context; + + @BeforeEach + public void setUp() { + final AssetManager assetManager = mock(AssetManager.class); + final SecretsManager secretsManager = mock(SecretsManager.class); + context = new StandardConnectorConfigurationContext(assetManager, secretsManager); + } + + private StepConfiguration toStepConfiguration(final Map stringProperties) { + final Map valueReferences = new HashMap<>(); + for (final Map.Entry entry : stringProperties.entrySet()) { + final String value = entry.getValue(); + valueReferences.put(entry.getKey(), value == null ? null : new StringLiteralValue(value)); + } + return new StepConfiguration(valueReferences); + } + + @Test + public void testSetPropertiesWithNoExistingConfigurations() { + final Map properties = new HashMap<>(); + properties.put("key1", "value1"); + properties.put("key2", "value2"); + + context.setProperties("step1", toStepConfiguration(properties)); + + assertEquals("value1", context.getProperty("step1", "key1").getValue()); + assertEquals("value2", context.getProperty("step1", "key2").getValue()); + } + + @Test + public void testSetPropertiesAddsNewProperties() { + final Map initialProperties = new HashMap<>(); + initialProperties.put("key1", "value1"); + context.setProperties("step1", toStepConfiguration(initialProperties)); + + final Map newProperties = new HashMap<>(); + newProperties.put("key3", "value3"); + context.setProperties("step1", toStepConfiguration(newProperties)); + + assertEquals("value1", context.getProperty("step1", "key1").getValue()); + assertEquals("value3", context.getProperty("step1", "key3").getValue()); + } + + @Test + public void testSetPropertiesMergesExistingPropertiesWithNewValues() { + final Map initialProperties = new HashMap<>(); + initialProperties.put("key1", "value1"); + initialProperties.put("key2", "value2"); + context.setProperties("step1", toStepConfiguration(initialProperties)); + + final Map updatedProperties = new HashMap<>(); + updatedProperties.put("key2", "updatedValue2"); + updatedProperties.put("key3", "value3"); + context.setProperties("step1", toStepConfiguration(updatedProperties)); + + assertEquals("value1", context.getProperty("step1", "key1").getValue()); + assertEquals("updatedValue2", context.getProperty("step1", "key2").getValue()); + assertEquals("value3", context.getProperty("step1", "key3").getValue()); + } + + @Test + public void testSetPropertiesRemovesKeyWhenValueIsNull() { + final Map initialProperties = new HashMap<>(); + initialProperties.put("key1", "value1"); + initialProperties.put("key2", "value2"); + initialProperties.put("key3", "value3"); + context.setProperties("step1", toStepConfiguration(initialProperties)); + + assertEquals("value2", context.getProperty("step1", "key2").getValue()); + + final Map updatedProperties = new HashMap<>(); + updatedProperties.put("key2", null); + context.setProperties("step1", toStepConfiguration(updatedProperties)); + + assertEquals("value1", context.getProperty("step1", "key1").getValue()); + assertNull(context.getProperty("step1", "key2").getValue()); + assertEquals("value3", context.getProperty("step1", "key3").getValue()); + } + + @Test + public void testSetPropertiesLeavesUnprovidedKeysAsIs() { + final Map initialProperties = new HashMap<>(); + initialProperties.put("key1", "value1"); + initialProperties.put("key2", "value2"); + initialProperties.put("key3", "value3"); + context.setProperties("step1", toStepConfiguration(initialProperties)); + + final Map updatedProperties = new HashMap<>(); + updatedProperties.put("key2", "updatedValue2"); + context.setProperties("step1", toStepConfiguration(updatedProperties)); + + assertEquals("value1", context.getProperty("step1", "key1").getValue()); + assertEquals("updatedValue2", context.getProperty("step1", "key2").getValue()); + assertEquals("value3", context.getProperty("step1", "key3").getValue()); + } + + @Test + public void testSetPropertiesForDifferentSteps() { + final Map step1Properties = new HashMap<>(); + step1Properties.put("key1", "value1"); + context.setProperties("step1", toStepConfiguration(step1Properties)); + + final Map step2Properties = new HashMap<>(); + step2Properties.put("key2", "value2"); + context.setProperties("step2", toStepConfiguration(step2Properties)); + + assertEquals("value1", context.getProperty("step1", "key1").getValue()); + assertNull(context.getProperty("step1", "key2").getValue()); + assertEquals("value2", context.getProperty("step2", "key2").getValue()); + assertNull(context.getProperty("step2", "key1").getValue()); + } + + @Test + public void testGetPropertyReturnsEmptyForNonExistentStep() { + final ConnectorPropertyValue propertyValue = context.getProperty("nonExistentStep", "someProperty"); + assertNull(propertyValue.getValue()); + } + + @Test + public void testGetPropertyReturnsEmptyForNonExistentProperty() { + final Map properties = new HashMap<>(); + properties.put("key1", "value1"); + context.setProperties("step1", toStepConfiguration(properties)); + + final ConnectorPropertyValue propertyValue = context.getProperty("step1", "nonExistentProperty"); + assertNull(propertyValue.getValue()); + } + + @Test + public void testComplexMergingScenario() { + final Map initialProps = new HashMap<>(); + initialProps.put("a", "1"); + initialProps.put("b", "2"); + initialProps.put("c", "3"); + initialProps.put("d", "4"); + initialProps.put("e", "5"); + initialProps.put("f", "6"); + context.setProperties("step1", toStepConfiguration(initialProps)); + + final Map updateProps = new HashMap<>(); + updateProps.put("b", null); + updateProps.put("c", "30"); + updateProps.put("g", "7"); + updateProps.put("h", "8"); + context.setProperties("step1", toStepConfiguration(updateProps)); + + assertEquals("1", context.getProperty("step1", "a").getValue()); + assertNull(context.getProperty("step1", "b").getValue()); + assertEquals("30", context.getProperty("step1", "c").getValue()); + assertEquals("4", context.getProperty("step1", "d").getValue()); + assertEquals("5", context.getProperty("step1", "e").getValue()); + assertEquals("6", context.getProperty("step1", "f").getValue()); + assertEquals("7", context.getProperty("step1", "g").getValue()); + assertEquals("8", context.getProperty("step1", "h").getValue()); + } + + @Test + public void testCloneHasCorrectResolvedValues() { + final Map step1Properties = new HashMap<>(); + step1Properties.put("key1", "value1"); + step1Properties.put("key2", "value2"); + step1Properties.put("key3", "value3"); + context.setProperties("step1", toStepConfiguration(step1Properties)); + + final Map step2Properties = new HashMap<>(); + step2Properties.put("keyA", "valueA"); + step2Properties.put("keyB", "valueB"); + context.setProperties("step2", toStepConfiguration(step2Properties)); + + final MutableConnectorConfigurationContext clonedContext = context.clone(); + + assertEquals("value1", clonedContext.getProperty("step1", "key1").getValue()); + assertEquals("value2", clonedContext.getProperty("step1", "key2").getValue()); + assertEquals("value3", clonedContext.getProperty("step1", "key3").getValue()); + assertEquals("valueA", clonedContext.getProperty("step2", "keyA").getValue()); + assertEquals("valueB", clonedContext.getProperty("step2", "keyB").getValue()); + assertNull(clonedContext.getProperty("step1", "nonExistent").getValue()); + assertNull(clonedContext.getProperty("nonExistentStep", "key1").getValue()); + } + + @Test + public void testResolvePropertyValuesResolvesSecretsThatWereInitiallyUnresolvable() { + final String providerId = "provider-1"; + final String providerName = "TestProvider"; + final String secretName = "mySecret"; + final String fullyQualifiedName = "TestProvider.mySecret"; + final String secretValue = "super-secret-value"; + + final SecretProvider secretProvider = mock(SecretProvider.class); + when(secretProvider.getProviderId()).thenReturn(providerId); + when(secretProvider.getProviderName()).thenReturn(providerName); + + final Secret secret = mock(Secret.class); + when(secret.getValue()).thenReturn(secretValue); + when(secretProvider.getSecrets(List.of(fullyQualifiedName))).thenReturn(List.of(secret)); + + final SecretsManager secretsManager = mock(SecretsManager.class); + when(secretsManager.getSecretProviders()).thenReturn(Collections.emptySet()); + + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorConfigurationContext testContext = new StandardConnectorConfigurationContext(assetManager, secretsManager); + + final SecretReference secretRef = new SecretReference(providerId, providerName, secretName, fullyQualifiedName); + final Map properties = new HashMap<>(); + properties.put("plainProp", new StringLiteralValue("plainValue")); + properties.put("secretProp", secretRef); + testContext.setProperties("authStep", new StepConfiguration(properties)); + + assertEquals("plainValue", testContext.getProperty("authStep", "plainProp").getValue()); + assertNull(testContext.getProperty("authStep", "secretProp").getValue()); + + when(secretsManager.getSecretProviders()).thenReturn(Set.of(secretProvider)); + testContext.resolvePropertyValues(); + + assertEquals("plainValue", testContext.getProperty("authStep", "plainProp").getValue()); + assertEquals(secretValue, testContext.getProperty("authStep", "secretProp").getValue()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorInitializationContext.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorInitializationContext.java new file mode 100644 index 000000000000..565ddb05f9c7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorInitializationContext.java @@ -0,0 +1,387 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.components.connector.components.ParameterValue; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.ComponentType; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.logging.ComponentLog; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestStandardConnectorInitializationContext { + + + @Test + public void testCreateParameterValuesSingleContext() { + final VersionedParameterContext context = new VersionedParameterContext(); + context.setName("ctx"); + context.setInheritedParameterContexts(List.of()); + + final Set parameters = new HashSet<>(); + parameters.add(createVersionedParameter("p1", "v1", false)); + parameters.add(createVersionedParameter("p2", "secret", true)); + context.setParameters(parameters); + + final List parameterValues = createParameterValues(List.of(context)); + assertEquals(2, parameterValues.size()); + + final Map byName = indexByName(parameterValues); + assertEquals("v1", byName.get("p1").getValue()); + assertFalse(byName.get("p1").isSensitive()); + assertEquals("secret", byName.get("p2").getValue()); + assertTrue(byName.get("p2").isSensitive()); + } + + @Test + public void testCreateParameterValuesInheritedPrecedenceOrder() { + // Inherited contexts listed highest precedence first; reverse traversal ensures last applied wins + final VersionedParameterContext high = new VersionedParameterContext(); + high.setName("high"); + high.setInheritedParameterContexts(List.of()); + high.setParameters(Set.of(createVersionedParameter("p", "H", false))); + + final VersionedParameterContext low = new VersionedParameterContext(); + low.setName("low"); + low.setInheritedParameterContexts(List.of()); + low.setParameters(Set.of(createVersionedParameter("p", "L", false))); + + final VersionedParameterContext child = new VersionedParameterContext(); + child.setName("child"); + child.setInheritedParameterContexts(List.of("high", "low")); + child.setParameters(Set.of()); + + final Collection contexts = List.of(child, high, low); + final List parameterValues = createParameterValues(contexts); + final Map byName = indexByName(parameterValues); + assertEquals(1, byName.size()); + assertEquals("H", byName.get("p").getValue()); + } + + @Test + public void testCreateParameterValuesLocalOverridesInherited() { + final VersionedParameterContext base = new VersionedParameterContext(); + base.setName("base"); + base.setInheritedParameterContexts(List.of()); + base.setParameters(Set.of( + createVersionedParameter("x", "1", false), + createVersionedParameter("y", "Y", false) + )); + + final VersionedParameterContext mid = new VersionedParameterContext(); + mid.setName("mid"); + mid.setInheritedParameterContexts(List.of("base")); + mid.setParameters(Set.of( + createVersionedParameter("x", "2", false) + )); + + final VersionedParameterContext top = new VersionedParameterContext(); + top.setName("top"); + top.setInheritedParameterContexts(List.of("mid")); + top.setParameters(Set.of( + createVersionedParameter("x", "3", false), + createVersionedParameter("z", "Z", true) + )); + + final List contexts = new ArrayList<>(); + contexts.add(top); + contexts.add(mid); + contexts.add(base); + + final List parameterValues = createParameterValues(contexts); + final Map byName = indexByName(parameterValues); + assertEquals(3, byName.size()); + assertEquals("3", byName.get("x").getValue()); + assertEquals("Y", byName.get("y").getValue()); + assertEquals("Z", byName.get("z").getValue()); + assertTrue(byName.get("z").isSensitive()); + } + + private VersionedParameter createVersionedParameter(final String name, final String value, final boolean sensitive) { + final VersionedParameter parameter = new VersionedParameter(); + parameter.setName(name); + parameter.setValue(value); + parameter.setSensitive(sensitive); + return parameter; + } + + private Map indexByName(final List values) { + final Map byName = new HashMap<>(); + for (final ParameterValue value : values) { + byName.put(value.getName(), value); + } + return byName; + } + + private List createParameterValues(final Collection contexts) { + final ConnectorParameterLookup parameterLookup = new ConnectorParameterLookup(contexts, null); + return parameterLookup.getParameterValues(); + } + + @Test + public void testResolveBundlesRequireExactBundle() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + final Bundle availableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")).thenReturn(List.of(availableBundle)); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", + "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + + final VersionedExternalFlow externalFlow = new VersionedExternalFlow(); + externalFlow.setFlowContents(group); + externalFlow.setParameterContexts(Map.of()); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.REQUIRE_EXACT_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("0.0.0-NONEXISTENT", processor.getBundle().getVersion(), "REQUIRE_EXACT_BUNDLE should not change the bundle"); + } + + @Test + public void testResolveBundlesResolveBundleWithSingleAvailable() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + final Bundle availableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")).thenReturn(List.of(availableBundle)); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("2.0.0", processor.getBundle().getVersion(), "RESOLVE_BUNDLE should resolve to the single available bundle"); + } + + @Test + public void testResolveBundlesResolveBundleWithMultipleAvailable() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + final Bundle availableBundle1 = createBundle("org.apache.nifi", "nifi-standard-nar", "1.0.0"); + final Bundle availableBundle2 = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")).thenReturn(List.of(availableBundle1, availableBundle2)); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("0.0.0-NONEXISTENT", processor.getBundle().getVersion(), "RESOLVE_BUNDLE should not change bundle when multiple are available"); + } + + @Test + public void testResolveBundlesResolveBundleWithNoneAvailable() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")).thenReturn(List.of()); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("0.0.0-NONEXISTENT", processor.getBundle().getVersion(), "RESOLVE_BUNDLE should not change bundle when none are available"); + } + + @Test + public void testResolveBundlesResolveNewestBundle() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + final Bundle newestBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")).thenReturn(List.of(createBundle("org.apache.nifi", "nifi-standard-nar", "1.0.0"), newestBundle)); + when(bundleLookup.getLatestBundle("org.apache.nifi.processors.standard.GenerateFlowFile")).thenReturn(Optional.of(newestBundle)); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_NEWEST_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("2.0.0", processor.getBundle().getVersion(), "RESOLVE_NEWEST_BUNDLE should resolve to the newest available bundle"); + } + + @Test + public void testResolveBundlesResolveNewestBundleWithNoneAvailable() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")) + .thenReturn(List.of()); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", + "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_NEWEST_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("0.0.0-NONEXISTENT", processor.getBundle().getVersion(), "RESOLVE_NEWEST_BUNDLE should not change bundle when none are available"); + } + + @Test + public void testResolveBundlesWithControllerService() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + final Bundle availableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.ssl.StandardSSLContextService")) + .thenReturn(List.of(availableBundle)); + + final VersionedProcessGroup group = createProcessGroupWithControllerService("SSLContextService", + "org.apache.nifi.ssl.StandardSSLContextService", unavailableBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_BUNDLE); + + final VersionedControllerService service = group.getControllerServices().iterator().next(); + assertEquals("2.0.0", service.getBundle().getVersion(), "RESOLVE_BUNDLE should resolve controller service bundle"); + } + + @Test + public void testResolveBundlesWithNestedProcessGroups() { + final Bundle unavailableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "0.0.0-NONEXISTENT"); + final Bundle availableBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")) + .thenReturn(List.of(availableBundle)); + + final VersionedProcessGroup childGroup = createProcessGroupWithProcessor("GenerateFlowFile", + "org.apache.nifi.processors.standard.GenerateFlowFile", unavailableBundle); + childGroup.setIdentifier("child-group"); + + final VersionedProcessGroup rootGroup = new VersionedProcessGroup(); + rootGroup.setIdentifier("root-group"); + rootGroup.setName("Root"); + rootGroup.setProcessors(new HashSet<>()); + rootGroup.setControllerServices(new HashSet<>()); + rootGroup.setProcessGroups(Set.of(childGroup)); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(rootGroup, BundleCompatibility.RESOLVE_BUNDLE); + + final VersionedProcessor processor = childGroup.getProcessors().iterator().next(); + assertEquals("2.0.0", processor.getBundle().getVersion(), "RESOLVE_BUNDLE should resolve bundles in nested process groups"); + } + + @Test + public void testResolveBundlesDoesNotChangeAvailableBundle() { + final Bundle specifiedBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "1.0.0"); + final Bundle newerBundle = createBundle("org.apache.nifi", "nifi-standard-nar", "2.0.0"); + + final ComponentBundleLookup bundleLookup = mock(ComponentBundleLookup.class); + when(bundleLookup.getAvailableBundles("org.apache.nifi.processors.standard.GenerateFlowFile")) + .thenReturn(List.of(specifiedBundle, newerBundle)); + + final VersionedProcessGroup group = createProcessGroupWithProcessor("GenerateFlowFile", + "org.apache.nifi.processors.standard.GenerateFlowFile", specifiedBundle); + + final StandardConnectorInitializationContext context = createContext(bundleLookup); + context.resolveBundles(group, BundleCompatibility.RESOLVE_NEWEST_BUNDLE); + + final VersionedProcessor processor = group.getProcessors().iterator().next(); + assertEquals("1.0.0", processor.getBundle().getVersion(), "Should not change bundle when it is already available"); + } + + private Bundle createBundle(final String group, final String artifact, final String version) { + final Bundle bundle = new Bundle(); + bundle.setGroup(group); + bundle.setArtifact(artifact); + bundle.setVersion(version); + return bundle; + } + + private VersionedProcessGroup createProcessGroupWithProcessor(final String name, final String type, final Bundle bundle) { + final VersionedProcessor processor = new VersionedProcessor(); + processor.setIdentifier("processor-1"); + processor.setName(name); + processor.setType(type); + processor.setBundle(bundle); + processor.setComponentType(ComponentType.PROCESSOR); + processor.setProperties(new HashMap<>()); + processor.setPropertyDescriptors(new HashMap<>()); + + final VersionedProcessGroup group = new VersionedProcessGroup(); + group.setIdentifier("test-group"); + group.setName("Test Group"); + group.setProcessors(new HashSet<>(Set.of(processor))); + group.setControllerServices(new HashSet<>()); + group.setProcessGroups(new HashSet<>()); + return group; + } + + private VersionedProcessGroup createProcessGroupWithControllerService(final String name, final String type, final Bundle bundle) { + final VersionedControllerService service = new VersionedControllerService(); + service.setIdentifier("service-1"); + service.setName(name); + service.setType(type); + service.setBundle(bundle); + service.setComponentType(ComponentType.CONTROLLER_SERVICE); + service.setProperties(new HashMap<>()); + service.setPropertyDescriptors(new HashMap<>()); + + final VersionedProcessGroup group = new VersionedProcessGroup(); + group.setIdentifier("test-group"); + group.setName("Test Group"); + group.setProcessors(new HashSet<>()); + group.setControllerServices(new HashSet<>(Set.of(service))); + group.setProcessGroups(new HashSet<>()); + return group; + } + + private StandardConnectorInitializationContext createContext(final ComponentBundleLookup bundleLookup) { + return new StandardConnectorInitializationContext.Builder() + .identifier("test-connector") + .name("Test Connector") + .componentLog(mock(ComponentLog.class)) + .secretsManager(mock(SecretsManager.class)) + .assetManager(mock(AssetManager.class)) + .componentBundleLookup(bundleLookup) + .build(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorNode.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorNode.java new file mode 100644 index 000000000000..d306ac45aa83 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorNode.java @@ -0,0 +1,848 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.FlowContextType; +import org.apache.nifi.components.connector.secrets.SecretsManager; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.engine.FlowEngine; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.util.MockComponentLog; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.Mock; +import org.mockito.MockitoAnnotations; + +import java.time.Duration; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicReference; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.anySet; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestStandardConnectorNode { + + private FlowEngine scheduler; + + @Mock + private ExtensionManager extensionManager; + @Mock + private ProcessGroup managedProcessGroup; + @Mock + private AssetManager assetManager; + @Mock + private SecretsManager secretsManager; + + private FlowContextFactory flowContextFactory; + + @BeforeEach + public void setUp() { + MockitoAnnotations.openMocks(this); + scheduler = new FlowEngine(1, "flow-engine"); + + when(managedProcessGroup.purge()).thenReturn(CompletableFuture.completedFuture(null)); + when(managedProcessGroup.getQueueSize()).thenReturn(new QueueSize(0, 0L)); + + flowContextFactory = new FlowContextFactory() { + @Override + public FrameworkFlowContext createActiveFlowContext(final String connectorId, final ComponentLog connectorLogger, final Bundle bundle) { + final MutableConnectorConfigurationContext activeConfigurationContext = new StandardConnectorConfigurationContext(assetManager, secretsManager); + final ProcessGroupFacadeFactory processGroupFacadeFactory = mock(ProcessGroupFacadeFactory.class); + final ParameterContextFacadeFactory parameterContextFacadeFactory = mock(ParameterContextFacadeFactory.class); + return new StandardFlowContext(managedProcessGroup, activeConfigurationContext, processGroupFacadeFactory, + parameterContextFacadeFactory, connectorLogger, FlowContextType.ACTIVE, bundle); + } + + @Override + public FrameworkFlowContext createWorkingFlowContext(final String connectorId, final ComponentLog connectorLogger, + final MutableConnectorConfigurationContext currentConfiguration, final Bundle bundle) { + + final ProcessGroupFacadeFactory processGroupFacadeFactory = mock(ProcessGroupFacadeFactory.class); + final ParameterContextFacadeFactory parameterContextFacadeFactory = mock(ParameterContextFacadeFactory.class); + + return new StandardFlowContext(managedProcessGroup, currentConfiguration, processGroupFacadeFactory, + parameterContextFacadeFactory, connectorLogger, FlowContextType.WORKING, bundle); + } + }; + } + + @AfterEach + public void teardown() { + if (scheduler != null) { + scheduler.close(); + } + } + + @Test + public void testStartFromStoppedState() throws Exception { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + final Future startFuture = connectorNode.start(scheduler); + + startFuture.get(5, TimeUnit.SECONDS); + + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + assertTrue(startFuture.isDone()); + assertFalse(startFuture.isCancelled()); + } + + @Test + public void testStopFromRunningState() throws Exception { + final StandardConnectorNode connectorNode = createConnectorNode(); + final Future startFuture = connectorNode.start(scheduler); + startFuture.get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + + final Future stopFuture = connectorNode.stop(scheduler); + stopFuture.get(5, TimeUnit.SECONDS); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + assertEquals(ConnectorState.STOPPED, connectorNode.getDesiredState()); + assertTrue(stopFuture.isDone()); + assertFalse(stopFuture.isCancelled()); + } + + @Test + public void testStartFutureCompletedOnlyWhenRunning() throws Exception { + final StandardConnectorNode connectorNode = createConnectorNode(); + final Future startFuture = connectorNode.start(scheduler); + + startFuture.get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + assertTrue(startFuture.isDone()); + } + + @Test + public void testStopFutureCompletedOnlyWhenStopped() throws Exception { + final StandardConnectorNode connectorNode = createConnectorNode(); + connectorNode.start(scheduler).get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + + final Future stopFuture = connectorNode.stop(scheduler); + stopFuture.get(5, TimeUnit.SECONDS); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + assertTrue(stopFuture.isDone()); + } + + @Test + @Timeout(value = 5, unit = TimeUnit.SECONDS) + public void testMultipleStartCallsReturnCompletedFutures() throws Exception { + final CountDownLatch startLatch = new CountDownLatch(1); + final BlockingConnector blockingConnector = new BlockingConnector(startLatch, new CountDownLatch(0), new CountDownLatch(0)); + final StandardConnectorNode connectorNode = createConnectorNode(blockingConnector); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + final Future startFuture1 = connectorNode.start(scheduler); + assertEquals(ConnectorState.STARTING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + + final Future startFuture2 = connectorNode.start(scheduler); + assertEquals(ConnectorState.STARTING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + + // Allow the connector to start + startLatch.countDown(); + startFuture1.get(5, TimeUnit.SECONDS); + startFuture2.get(5, TimeUnit.SECONDS); + + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + + assertTrue(startFuture1.isDone()); + assertTrue(startFuture2.isDone()); + } + + @Test + public void testVerifyCanDeleteWhenStopped() throws FlowUpdateException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + connectorNode.verifyCanDelete(); + } + + @Test + public void testCannotDeleteWhenRunning() throws Exception { + final StandardConnectorNode connectorNode = createConnectorNode(); + connectorNode.start(scheduler).get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + + assertThrows(IllegalStateException.class, connectorNode::verifyCanDelete); + } + + @Test + public void testVerifyCanStartWhenStopped() throws FlowUpdateException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + connectorNode.verifyCanStart(); + } + + @Test + public void testStartAlreadyRunningReturnsImmediately() throws Exception { + final StandardConnectorNode connectorNode = createConnectorNode(); + connectorNode.start(scheduler).get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + + final Future startFuture = connectorNode.start(scheduler); + assertTrue(startFuture.isDone()); + + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + } + + @Test + public void testStopAlreadyStoppedReturnsImmediately() throws FlowUpdateException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + final Future stopFuture = connectorNode.stop(scheduler); + assertTrue(stopFuture.isDone()); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + } + + @Test + public void testStartWhileStoppingQueuesStartFuture() throws Exception { + final CountDownLatch stopLatch = new CountDownLatch(1); + final BlockingConnector blockingConnector = new BlockingConnector(new CountDownLatch(0), stopLatch, new CountDownLatch(0)); + final StandardConnectorNode connectorNode = createConnectorNode(blockingConnector); + + connectorNode.start(scheduler).get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + + final Future stopFuture = connectorNode.stop(scheduler); + assertEquals(ConnectorState.STOPPING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.STOPPED, connectorNode.getDesiredState()); + + final Future startFuture = connectorNode.start(scheduler); + assertEquals(ConnectorState.STOPPING, connectorNode.getCurrentState()); + assertEquals(ConnectorState.RUNNING, connectorNode.getDesiredState()); + + stopLatch.countDown(); + + stopFuture.get(5, TimeUnit.SECONDS); + startFuture.get(5, TimeUnit.SECONDS); + + assertEquals(ConnectorState.RUNNING, connectorNode.getCurrentState()); + assertTrue(stopFuture.isDone()); + assertTrue(startFuture.isDone()); + } + + @Test + public void testCannotDeleteWhenStarting() throws Exception { + // Use a slow-starting connector to test deletion during STARTING state + final CountDownLatch startLatch = new CountDownLatch(1); + final BlockingConnector blockingConnector = new BlockingConnector(startLatch, new CountDownLatch(0), new CountDownLatch(0)); + final StandardConnectorNode slowNode = createConnectorNode(blockingConnector); + + // Start the connector - this will take time + final Future startFuture = slowNode.start(scheduler); + + // While starting, verify we cannot delete + assertEquals(ConnectorState.STARTING, slowNode.getCurrentState()); + + assertThrows(IllegalStateException.class, slowNode::verifyCanDelete); + + // Wait for start to complete + startLatch.countDown(); + startFuture.get(5, TimeUnit.SECONDS); + assertEquals(ConnectorState.RUNNING, slowNode.getCurrentState()); + } + + @Test + public void testSetConfigurationWhenStopped() throws FlowUpdateException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + assertEquals(ConnectorState.STOPPED, connectorNode.getDesiredState()); + + final ConnectorConfiguration newConfiguration = createTestConfiguration(); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("testGroup", createStepConfiguration()); + connectorNode.applyUpdate(); + + assertEquals(newConfiguration, connectorNode.getActiveFlowContext().getConfigurationContext().toConnectorConfiguration()); + } + + @Test + public void testSetConfigurationWithPropertyChanges() throws FlowUpdateException, ExecutionException, InterruptedException, TimeoutException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("step1", createStepConfiguration(Map.of("prop1", "value1"))); + connectorNode.applyUpdate(); + + final ConnectorConfiguration newConfiguration = createTestConfiguration("step1", "prop1", "value2"); + + connectorNode.stop(scheduler).get(5, TimeUnit.SECONDS); + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("step1", createStepConfiguration(Map.of("prop1", "value2"))); + connectorNode.applyUpdate(); + assertEquals(newConfiguration, connectorNode.getActiveFlowContext().getConfigurationContext().toConnectorConfiguration()); + } + + @Test + public void testSetConfigurationWithNewConfigurationStep() throws FlowUpdateException, ExecutionException, InterruptedException, TimeoutException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("configurationStep1", createStepConfiguration(Map.of("prop1", "value1"))); + connectorNode.applyUpdate(); + + final ConnectorConfiguration newConfiguration = createTestConfigurationWithMultipleGroups(); + + // Wait for Connector to fully stop + connectorNode.stop(scheduler).get(5, TimeUnit.SECONDS); + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("configurationStep1", createStepConfiguration(Map.of("prop1", "value1"))); + connectorNode.setConfiguration("configurationStep2", createStepConfiguration(Map.of("prop2", "value2"))); + connectorNode.applyUpdate(); + + assertEquals(newConfiguration, connectorNode.getActiveFlowContext().getConfigurationContext().toConnectorConfiguration()); + } + + @Test + public void testSetConfigurationWithRemovedConfigurationStep() throws FlowUpdateException, ExecutionException, InterruptedException, TimeoutException { + final StandardConnectorNode connectorNode = createConnectorNode(); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("configurationStep1", createStepConfiguration(Map.of("prop1", "value1"))); + connectorNode.setConfiguration("configurationStep2", createStepConfiguration(Map.of("prop2", "value2"))); + connectorNode.applyUpdate(); + + connectorNode.stop(scheduler).get(5, TimeUnit.SECONDS); + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("configurationStep1", createStepConfiguration(Map.of("prop1", "value1"))); + connectorNode.applyUpdate(); + + final Set expectedSteps = Set.of( + new NamedStepConfiguration("configurationStep1", new StepConfiguration(Map.of("prop1", new StringLiteralValue("value1")))), + new NamedStepConfiguration("configurationStep2", new StepConfiguration(Map.of("prop2", new StringLiteralValue("value2")))) + ); + final ConnectorConfiguration expectedConfiguration = new ConnectorConfiguration(expectedSteps); + assertEquals(expectedConfiguration, connectorNode.getActiveFlowContext().getConfigurationContext().toConnectorConfiguration()); + } + + @Test + public void testSetConfigurationCallsOnConfigured() throws FlowUpdateException { + final TrackingConnector trackingConnector = new TrackingConnector(); + final StandardConnectorNode connectorNode = createConnectorNode(trackingConnector); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("testGroup", createStepConfiguration()); + connectorNode.applyUpdate(); + } + + @Test + public void testSetConfigurationCallsOnPropertyGroupConfiguredForChangedConfigurationSteps() throws FlowUpdateException, ExecutionException, InterruptedException, TimeoutException { + final TrackingConnector trackingConnector = new TrackingConnector(); + final StandardConnectorNode connectorNode = createConnectorNode(trackingConnector); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("configurationStep1", createStepConfiguration(Map.of("prop1", "value1"))); + connectorNode.applyUpdate(); + trackingConnector.reset(); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + connectorNode.setConfiguration("configurationStep1", createStepConfiguration(Map.of("prop1", "value2"))); + connectorNode.applyUpdate(); + + assertTrue(trackingConnector.wasOnPropertyGroupConfiguredCalled("configurationStep1")); + } + + @Test + public void testVerifyConfigurationStepSurfacesInvalidValidationResults() throws FlowUpdateException { + final ValidationFailingConnector validationFailingConnector = new ValidationFailingConnector(); + final StandardConnectorNode connectorNode = createConnectorNode(validationFailingConnector); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.transitionStateForUpdating(); + connectorNode.prepareForUpdate(); + final List results = connectorNode.verifyConfigurationStep("testStep", new StepConfiguration(Map.of("testProperty", new StringLiteralValue("invalidValue")))); + + assertFalse(results.isEmpty()); + final List failedResults = results.stream() + .filter(result -> result.getOutcome() == ConfigVerificationResult.Outcome.FAILED) + .toList(); + assertFalse(failedResults.isEmpty()); + + final ConfigVerificationResult failedResult = failedResults.getFirst(); + assertEquals("Property Validation - Test Property", failedResult.getVerificationStepName()); + assertEquals("Test Property", failedResult.getSubject()); + assertEquals("The property value is invalid", failedResult.getExplanation()); + } + + @Test + @Timeout(value = 5, unit = TimeUnit.SECONDS) + public void testDrainFlowFilesTransitionsStateToDraining() throws FlowUpdateException { + final CompletableFuture drainCompletionFuture = new CompletableFuture<>(); + final DrainBlockingConnector drainBlockingConnector = new DrainBlockingConnector(drainCompletionFuture); + final StandardConnectorNode connectorNode = createConnectorNode(drainBlockingConnector); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + final Future drainFuture = connectorNode.drainFlowFiles(); + + assertEquals(ConnectorState.DRAINING, connectorNode.getCurrentState()); + assertFalse(drainFuture.isDone()); + + drainCompletionFuture.complete(null); + + try { + drainFuture.get(2, TimeUnit.SECONDS); + } catch (final Exception e) { + throw new RuntimeException("Drain future failed to complete", e); + } + + assertTrue(drainFuture.isDone()); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + } + + @Test + @Timeout(value = 5, unit = TimeUnit.SECONDS) + public void testDrainFlowFilesFutureDoesNotCompleteUntilDrainFinishes() throws FlowUpdateException, InterruptedException { + final CompletableFuture drainCompletionFuture = new CompletableFuture<>(); + final DrainBlockingConnector drainBlockingConnector = new DrainBlockingConnector(drainCompletionFuture); + final StandardConnectorNode connectorNode = createConnectorNode(drainBlockingConnector); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + final Future drainFuture = connectorNode.drainFlowFiles(); + assertEquals(ConnectorState.DRAINING, connectorNode.getCurrentState()); + + Thread.sleep(200); + assertFalse(drainFuture.isDone()); + assertEquals(ConnectorState.DRAINING, connectorNode.getCurrentState()); + + drainCompletionFuture.complete(null); + + try { + drainFuture.get(2, TimeUnit.SECONDS); + } catch (final Exception e) { + throw new RuntimeException("Drain future failed to complete", e); + } + + assertTrue(drainFuture.isDone()); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + } + + @Test + @Timeout(value = 5, unit = TimeUnit.SECONDS) + public void testDrainFlowFilesStateTransitionsBackToStoppedOnCompletion() throws FlowUpdateException { + final CompletableFuture drainCompletionFuture = new CompletableFuture<>(); + final DrainBlockingConnector drainBlockingConnector = new DrainBlockingConnector(drainCompletionFuture); + final StandardConnectorNode connectorNode = createConnectorNode(drainBlockingConnector); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + assertEquals(ConnectorState.STOPPED, connectorNode.getDesiredState()); + + final Future drainFuture = connectorNode.drainFlowFiles(); + assertEquals(ConnectorState.DRAINING, connectorNode.getCurrentState()); + + drainCompletionFuture.complete(null); + + try { + drainFuture.get(2, TimeUnit.SECONDS); + } catch (final Exception e) { + throw new RuntimeException("Drain future failed to complete", e); + } + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + assertEquals(ConnectorState.STOPPED, connectorNode.getDesiredState()); + } + + @Test + @Timeout(value = 10, unit = TimeUnit.SECONDS) + public void testCancelDrainFlowFilesInterruptsConnector() throws Exception { + final BusyLoopDrainConnector busyLoopConnector = new BusyLoopDrainConnector(); + final StandardConnectorNode connectorNode = createConnectorNode(busyLoopConnector); + + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + + connectorNode.drainFlowFiles(); + + assertTrue(busyLoopConnector.awaitDrainStarted(2, TimeUnit.SECONDS)); + assertEquals(ConnectorState.DRAINING, connectorNode.getCurrentState()); + + Thread.sleep(1000); + + assertEquals(ConnectorState.DRAINING, connectorNode.getCurrentState()); + assertFalse(busyLoopConnector.wasInterrupted()); + assertFalse(busyLoopConnector.wasStopCalled()); + + connectorNode.cancelDrainFlowFiles(); + + assertTrue(busyLoopConnector.awaitDrainCompleted(2, TimeUnit.SECONDS)); + assertTrue(busyLoopConnector.wasInterrupted()); + assertTrue(busyLoopConnector.wasStopCalled()); + assertEquals(ConnectorState.STOPPED, connectorNode.getCurrentState()); + } + + private StandardConnectorNode createConnectorNode() throws FlowUpdateException { + final SleepingConnector sleepingConnector = new SleepingConnector(Duration.ofMillis(1)); + return createConnectorNode(sleepingConnector); + } + + private StandardConnectorNode createConnectorNode(final Connector connector) throws FlowUpdateException { + final ConnectorStateTransition stateTransition = new StandardConnectorStateTransition("TestConnectorNode"); + final ConnectorValidationTrigger validationTrigger = new SynchronousConnectorValidationTrigger(); + final StandardConnectorNode node = new StandardConnectorNode( + "test-connector-id", + mock(FlowManager.class), + extensionManager, + null, + createConnectorDetails(connector), + "TestConnector", + connector.getClass().getCanonicalName(), + new StandardConnectorConfigurationContext(assetManager, secretsManager), + stateTransition, + flowContextFactory, + validationTrigger, + false); + + // mock secrets manager + final SecretsManager secretsManager = mock(SecretsManager.class); + when(secretsManager.getAllSecrets()).thenReturn(List.of()); + when(secretsManager.getSecrets(anySet())).thenReturn(Collections.emptyMap()); + + final FrameworkConnectorInitializationContext initializationContext = mock(FrameworkConnectorInitializationContext.class); + when(initializationContext.getSecretsManager()).thenReturn(secretsManager); + + node.initializeConnector(initializationContext); + node.loadInitialFlow(); + return node; + } + + private static class SynchronousConnectorValidationTrigger implements ConnectorValidationTrigger { + @Override + public void triggerAsync(final ConnectorNode connector) { + trigger(connector); + } + + @Override + public void trigger(final ConnectorNode connector) { + connector.performValidation(); + } + } + + private ConnectorDetails createConnectorDetails(final Connector connector) { + final ComponentLog componentLog = new MockComponentLog("TestConnector", connector); + final BundleCoordinate bundleCoordinate = new BundleCoordinate("org.apache.nifi", "test-standard-connector-node", "1.0.0"); + return new ConnectorDetails(connector, bundleCoordinate, componentLog); + } + + private StepConfiguration createStepConfiguration() { + return createStepConfiguration(Map.of("testProperty", "testValue")); + } + + private StepConfiguration createStepConfiguration(final Map properties) { + final Map valueReferences = new java.util.HashMap<>(); + for (final Map.Entry entry : properties.entrySet()) { + valueReferences.put(entry.getKey(), new StringLiteralValue(entry.getValue())); + } + return new StepConfiguration(valueReferences); + } + + private ConnectorConfiguration createTestConfiguration() { + return createTestConfiguration("testGroup", "testProperty", "testValue"); + } + + private ConnectorConfiguration createTestConfiguration(final String configurationStepName, final String propertyName, final String propertyValue) { + final StepConfiguration stepConfig = new StepConfiguration(Map.of(propertyName, new StringLiteralValue(propertyValue))); + final NamedStepConfiguration configurationStepConfiguration = new NamedStepConfiguration(configurationStepName, stepConfig); + return new ConnectorConfiguration(Set.of(configurationStepConfiguration)); + } + + private ConnectorConfiguration createTestConfigurationWithMultipleGroups() { + final StepConfiguration firstStepConfig = new StepConfiguration(Map.of("prop1", new StringLiteralValue("value1"))); + final NamedStepConfiguration firstConfigurationStepConfiguration = new NamedStepConfiguration("configurationStep1", firstStepConfig); + + final StepConfiguration secondStepConfig = new StepConfiguration(Map.of("prop2", new StringLiteralValue("value2"))); + final NamedStepConfiguration secondConfigurationStepConfiguration = new NamedStepConfiguration("configurationStep2", secondStepConfig); + + return new ConnectorConfiguration(Set.of(firstConfigurationStepConfiguration, secondConfigurationStepConfiguration)); + } + + /** + * Test connector that tracks method calls for verification + */ + private static class TrackingConnector extends AbstractConnector { + private final Set onConfigurationStepConfiguredCalls = new HashSet<>(); + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + onConfigurationStepConfiguredCalls.add(stepName); + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + public boolean wasOnPropertyGroupConfiguredCalled(final String stepName) { + return onConfigurationStepConfiguredCalls.contains(stepName); + } + + public void reset() { + onConfigurationStepConfiguredCalls.clear(); + } + } + + /** + * Test connector that returns invalid validation results from validateConfigurationStep + */ + private static class ValidationFailingConnector extends AbstractConnector { + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public List validateConfigurationStep(final ConfigurationStep configurationStep, final ConnectorConfigurationContext connectorConfigurationContext, + final ConnectorValidationContext connectorValidationContext) { + + final ValidationResult invalidResult = new ValidationResult.Builder() + .subject("Test Property") + .valid(false) + .explanation("The property value is invalid") + .build(); + return List.of(invalidResult); + } + + @Override + public List getConfigurationSteps() { + final ConfigurationStep testStep = new ConfigurationStep.Builder() + .name("testStep") + .build(); + return List.of(testStep); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + } + + /** + * Test connector that allows control over when drainFlowFiles completes via a CompletableFuture + */ + private static class DrainBlockingConnector extends AbstractConnector { + private final CompletableFuture drainCompletionFuture; + + public DrainBlockingConnector(final CompletableFuture drainCompletionFuture) { + this.drainCompletionFuture = drainCompletionFuture; + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + return drainCompletionFuture; + } + } + + /** + * Test connector that uses a busy loop for draining that never completes naturally, + * but can be interrupted via cancellation. + */ + private static class BusyLoopDrainConnector extends AbstractConnector { + private volatile boolean interrupted = false; + private volatile boolean stopCalled = false; + private final AtomicReference drainThreadRef = new AtomicReference<>(); + private final CountDownLatch drainStartedLatch = new CountDownLatch(1); + private final CountDownLatch drainCompletedLatch = new CountDownLatch(1); + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public void stop(final FlowContext activeContext) { + stopCalled = true; + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + final CompletableFuture future = new CompletableFuture<>(); + + final Thread drainThread = new Thread(() -> { + drainStartedLatch.countDown(); + try { + while (!Thread.currentThread().isInterrupted()) { + // Busy loop that never completes naturally + } + } finally { + interrupted = Thread.currentThread().isInterrupted(); + drainCompletedLatch.countDown(); + } + }); + + drainThreadRef.set(drainThread); + + future.whenComplete((result, throwable) -> { + final Thread thread = drainThreadRef.get(); + if (thread != null) { + thread.interrupt(); + } + }); + + drainThread.start(); + + return future; + } + + public boolean awaitDrainStarted(final long timeout, final TimeUnit unit) throws InterruptedException { + return drainStartedLatch.await(timeout, unit); + } + + public boolean awaitDrainCompleted(final long timeout, final TimeUnit unit) throws InterruptedException { + return drainCompletedLatch.await(timeout, unit); + } + + public boolean wasInterrupted() { + return interrupted; + } + + public boolean wasStopCalled() { + return stopCalled; + } + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorRepository.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorRepository.java new file mode 100644 index 000000000000..4089b274819e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/TestStandardConnectorRepository.java @@ -0,0 +1,877 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.asset.AssetManager; +import org.apache.nifi.flow.VersionedConfigurationStep; +import org.apache.nifi.flow.VersionedConnectorValueReference; +import org.apache.nifi.nar.ExtensionManager; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNotSame; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.timeout; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +public class TestStandardConnectorRepository { + + @Test + public void testAddAndGetConnectors() { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + + final ConnectorNode connector1 = mock(ConnectorNode.class); + final ConnectorNode connector2 = mock(ConnectorNode.class); + when(connector1.getIdentifier()).thenReturn("connector-1"); + when(connector2.getIdentifier()).thenReturn("connector-2"); + + repository.addConnector(connector1); + repository.addConnector(connector2); + + final List connectors = repository.getConnectors(); + assertEquals(2, connectors.size()); + assertTrue(connectors.contains(connector1)); + assertTrue(connectors.contains(connector2)); + } + + @Test + public void testRemoveConnector() { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + + final ConnectorRepositoryInitializationContext initContext = mock(ConnectorRepositoryInitializationContext.class); + final ExtensionManager extensionManager = mock(ExtensionManager.class); + when(initContext.getExtensionManager()).thenReturn(extensionManager); + repository.initialize(initContext); + + final Connector mockConnector = mock(Connector.class); + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + when(connector.getConnector()).thenReturn(mockConnector); + + repository.addConnector(connector); + repository.removeConnector("connector-1"); + + assertEquals(0, repository.getConnectors().size()); + assertNull(repository.getConnector("connector-1")); + } + + @Test + public void testRestoreConnector() { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + + repository.restoreConnector(connector); + assertEquals(1, repository.getConnectors().size()); + assertEquals(connector, repository.getConnector("connector-1")); + } + + @Test + public void testGetConnectorsReturnsNewListInstances() { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + + final ConnectorNode connector1 = mock(ConnectorNode.class); + final ConnectorNode connector2 = mock(ConnectorNode.class); + when(connector1.getIdentifier()).thenReturn("connector-1"); + when(connector2.getIdentifier()).thenReturn("connector-2"); + + repository.addConnector(connector1); + repository.addConnector(connector2); + + final List connectors1 = repository.getConnectors(); + final List connectors2 = repository.getConnectors(); + + assertEquals(2, connectors1.size()); + assertEquals(2, connectors2.size()); + assertEquals(connectors1, connectors2); + assertNotSame(connectors1, connectors2); + } + + @Test + public void testAddConnectorWithDuplicateIdReplaces() { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + + final ConnectorNode connector1 = mock(ConnectorNode.class); + final ConnectorNode connector2 = mock(ConnectorNode.class); + when(connector1.getIdentifier()).thenReturn("same-id"); + when(connector2.getIdentifier()).thenReturn("same-id"); + + repository.addConnector(connector1); + repository.addConnector(connector2); + + final List connectors = repository.getConnectors(); + assertEquals(1, connectors.size()); + assertEquals(connector2, repository.getConnector("same-id")); + } + + @Test + public void testDiscardWorkingConfigurationPreservesWorkingAssets() { + final String connectorId = "test-connector"; + final String activeAssetId = "active-asset-id"; + final String workingAssetId = "working-asset-id"; + final String unreferencedAssetId = "unreferenced-asset-id"; + + final StandardConnectorRepository repository = new StandardConnectorRepository(); + + final AssetManager assetManager = mock(AssetManager.class); + final ConnectorRepositoryInitializationContext initContext = mock(ConnectorRepositoryInitializationContext.class); + when(initContext.getExtensionManager()).thenReturn(mock(ExtensionManager.class)); + when(initContext.getAssetManager()).thenReturn(assetManager); + repository.initialize(initContext); + + final Asset activeAsset = mock(Asset.class); + when(activeAsset.getIdentifier()).thenReturn(activeAssetId); + when(activeAsset.getName()).thenReturn("active-asset.jar"); + + final Asset workingAsset = mock(Asset.class); + when(workingAsset.getIdentifier()).thenReturn(workingAssetId); + when(workingAsset.getName()).thenReturn("working-asset.jar"); + + final Asset unreferencedAsset = mock(Asset.class); + when(unreferencedAsset.getIdentifier()).thenReturn(unreferencedAssetId); + when(unreferencedAsset.getName()).thenReturn("unreferenced-asset.jar"); + + final MutableConnectorConfigurationContext activeConfigContext = mock(MutableConnectorConfigurationContext.class); + final StepConfiguration activeStepConfig = new StepConfiguration( + Map.of("prop1", new AssetReference(Set.of(activeAssetId))) + ); + final ConnectorConfiguration activeConfig = new ConnectorConfiguration( + Set.of(new NamedStepConfiguration("step1", activeStepConfig)) + ); + when(activeConfigContext.toConnectorConfiguration()).thenReturn(activeConfig); + + final MutableConnectorConfigurationContext workingConfigContext = mock(MutableConnectorConfigurationContext.class); + final StepConfiguration workingStepConfig = new StepConfiguration( + Map.of("prop1", new AssetReference(Set.of(workingAssetId))) + ); + final ConnectorConfiguration workingConfig = new ConnectorConfiguration( + Set.of(new NamedStepConfiguration("step1", workingStepConfig)) + ); + when(workingConfigContext.toConnectorConfiguration()).thenReturn(workingConfig); + + final FrameworkFlowContext activeFlowContext = mock(FrameworkFlowContext.class); + when(activeFlowContext.getConfigurationContext()).thenReturn(activeConfigContext); + + final FrameworkFlowContext workingFlowContext = mock(FrameworkFlowContext.class); + when(workingFlowContext.getConfigurationContext()).thenReturn(workingConfigContext); + + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn(connectorId); + when(connector.getActiveFlowContext()).thenReturn(activeFlowContext); + when(connector.getWorkingFlowContext()).thenReturn(workingFlowContext); + + when(assetManager.getAssets(connectorId)).thenReturn(List.of(activeAsset, workingAsset, unreferencedAsset)); + + repository.addConnector(connector); + repository.discardWorkingConfiguration(connector); + + verify(assetManager, never()).deleteAsset(activeAssetId); + verify(assetManager, never()).deleteAsset(workingAssetId); + verify(assetManager).deleteAsset(unreferencedAssetId); + } + + @Test + public void testGetConnectorWithProviderOverridesWorkingConfig() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final MutableConnectorConfigurationContext workingConfigContext = mock(MutableConnectorConfigurationContext.class); + final ConnectorNode connector = createConnectorNodeWithWorkingConfig("connector-1", "Original Name", workingConfigContext); + repository.addConnector(connector); + + final ConnectorWorkingConfiguration externalConfig = new ConnectorWorkingConfiguration(); + externalConfig.setName("External Name"); + final VersionedConfigurationStep externalStep = createVersionedStep("step1", Map.of("prop1", createStringLiteralRef("external-value"))); + externalConfig.setWorkingFlowConfiguration(List.of(externalStep)); + when(provider.load("connector-1")).thenReturn(Optional.of(externalConfig)); + + final ConnectorNode result = repository.getConnector("connector-1"); + + assertNotNull(result); + verify(connector).setName("External Name"); + verify(workingConfigContext).replaceProperties(eq("step1"), any(StepConfiguration.class)); + } + + @Test + public void testGetConnectorWithProviderReturnsEmpty() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createSimpleConnectorNode("connector-1", "Original Name"); + repository.addConnector(connector); + + when(provider.load("connector-1")).thenReturn(Optional.empty()); + + final ConnectorNode result = repository.getConnector("connector-1"); + + assertNotNull(result); + verify(connector, never()).setName(anyString()); + } + + @Test + public void testGetConnectorWithProviderThrowsException() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createSimpleConnectorNode("connector-1", "Original Name"); + repository.addConnector(connector); + + when(provider.load("connector-1")).thenThrow(new ConnectorConfigurationProviderException("Provider failure")); + + assertThrows(ConnectorConfigurationProviderException.class, () -> repository.getConnector("connector-1")); + verify(connector, never()).setName(anyString()); + } + + @Test + public void testGetConnectorWithNullProvider() { + final StandardConnectorRepository repository = createRepositoryWithProvider(null); + + final ConnectorNode connector = createSimpleConnectorNode("connector-1", "Original Name"); + repository.addConnector(connector); + + final ConnectorNode result = repository.getConnector("connector-1"); + + assertNotNull(result); + verify(connector, never()).setName(anyString()); + } + + @Test + public void testGetConnectorsWithProviderOverrides() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final MutableConnectorConfigurationContext workingConfig1 = mock(MutableConnectorConfigurationContext.class); + final MutableConnectorConfigurationContext workingConfig2 = mock(MutableConnectorConfigurationContext.class); + final ConnectorNode connector1 = createConnectorNodeWithWorkingConfig("connector-1", "Name 1", workingConfig1); + final ConnectorNode connector2 = createConnectorNodeWithWorkingConfig("connector-2", "Name 2", workingConfig2); + repository.addConnector(connector1); + repository.addConnector(connector2); + + final ConnectorWorkingConfiguration externalConfig1 = new ConnectorWorkingConfiguration(); + externalConfig1.setName("External Name 1"); + externalConfig1.setWorkingFlowConfiguration(List.of()); + when(provider.load("connector-1")).thenReturn(Optional.of(externalConfig1)); + when(provider.load("connector-2")).thenReturn(Optional.empty()); + + final List results = repository.getConnectors(); + + assertEquals(2, results.size()); + verify(connector1).setName("External Name 1"); + verify(connector2, never()).setName(anyString()); + } + + @Test + public void testConfigureConnectorSavesToProviderBeforeModifyingNode() throws FlowUpdateException { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createConnectorNodeWithEmptyWorkingConfig("connector-1", "Test Connector"); + repository.addConnector(connector); + + when(provider.load("connector-1")).thenReturn(Optional.empty()); + + final StepConfiguration incomingConfig = new StepConfiguration(Map.of("prop1", new StringLiteralValue("new-value"))); + repository.configureConnector(connector, "step1", incomingConfig); + + final ArgumentCaptor configCaptor = ArgumentCaptor.forClass(ConnectorWorkingConfiguration.class); + verify(provider).save(eq("connector-1"), configCaptor.capture()); + + final ConnectorWorkingConfiguration savedConfig = configCaptor.getValue(); + assertNotNull(savedConfig); + assertEquals("Test Connector", savedConfig.getName()); + + final List savedSteps = savedConfig.getWorkingFlowConfiguration(); + assertNotNull(savedSteps); + final VersionedConfigurationStep savedStep = savedSteps.stream() + .filter(s -> "step1".equals(s.getName())).findFirst().orElse(null); + assertNotNull(savedStep); + assertEquals("STRING_LITERAL", savedStep.getProperties().get("prop1").getValueType()); + assertEquals("new-value", savedStep.getProperties().get("prop1").getValue()); + + verify(connector).setConfiguration("step1", incomingConfig); + } + + @Test + public void testConfigureConnectorProviderSaveFailsDoesNotModifyNode() throws FlowUpdateException { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createConnectorNodeWithEmptyWorkingConfig("connector-1", "Test Connector"); + repository.addConnector(connector); + + when(provider.load("connector-1")).thenReturn(Optional.empty()); + doThrow(new RuntimeException("Save failed")).when(provider).save(anyString(), any(ConnectorWorkingConfiguration.class)); + + final StepConfiguration incomingConfig = new StepConfiguration(Map.of("prop1", new StringLiteralValue("new-value"))); + + assertThrows(RuntimeException.class, () -> repository.configureConnector(connector, "step1", incomingConfig)); + + verify(connector, never()).setConfiguration(anyString(), any(StepConfiguration.class)); + } + + @Test + public void testConfigureConnectorMergesPartialStepConfig() throws FlowUpdateException { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createConnectorNodeWithEmptyWorkingConfig("connector-1", "Test Connector"); + repository.addConnector(connector); + + final VersionedConfigurationStep existingStep = createVersionedStep("step1", + Map.of("propA", createStringLiteralRef("old-A"), "propB", createStringLiteralRef("old-B"))); + final ConnectorWorkingConfiguration existingConfig = new ConnectorWorkingConfiguration(); + existingConfig.setName("Test Connector"); + existingConfig.setWorkingFlowConfiguration(new ArrayList<>(List.of(existingStep))); + when(provider.load("connector-1")).thenReturn(Optional.of(existingConfig)); + + final Map incomingProps = new HashMap<>(); + incomingProps.put("propA", new StringLiteralValue("new-A")); + incomingProps.put("propC", new StringLiteralValue("new-C")); + final StepConfiguration incomingConfig = new StepConfiguration(incomingProps); + + repository.configureConnector(connector, "step1", incomingConfig); + + final ArgumentCaptor configCaptor = ArgumentCaptor.forClass(ConnectorWorkingConfiguration.class); + verify(provider).save(eq("connector-1"), configCaptor.capture()); + + final ConnectorWorkingConfiguration savedConfig = configCaptor.getValue(); + final VersionedConfigurationStep savedStep = savedConfig.getWorkingFlowConfiguration().stream() + .filter(s -> "step1".equals(s.getName())).findFirst().orElse(null); + assertNotNull(savedStep); + + final Map savedProps = savedStep.getProperties(); + assertEquals("new-A", savedProps.get("propA").getValue()); + assertEquals("old-B", savedProps.get("propB").getValue()); + assertEquals("new-C", savedProps.get("propC").getValue()); + + verify(connector).setConfiguration("step1", incomingConfig); + } + + @Test + public void testDiscardWorkingConfigurationCallsProviderDiscard() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createConnectorNodeWithEmptyWorkingConfig("connector-1", "Test Connector"); + repository.addConnector(connector); + + repository.discardWorkingConfiguration(connector); + + verify(provider).discard("connector-1"); + verify(provider, never()).save(anyString(), any(ConnectorWorkingConfiguration.class)); + verify(connector).discardWorkingConfiguration(); + } + + @Test + public void testDiscardWorkingConfigurationProviderThrowsException() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createConnectorNodeWithEmptyWorkingConfig("connector-1", "Test Connector"); + repository.addConnector(connector); + + doThrow(new ConnectorConfigurationProviderException("Discard failed")).when(provider).discard("connector-1"); + + assertThrows(ConnectorConfigurationProviderException.class, () -> repository.discardWorkingConfiguration(connector)); + } + + @Test + public void testRemoveConnectorCallsProviderDelete() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final Connector mockConnector = mock(Connector.class); + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + when(connector.getConnector()).thenReturn(mockConnector); + repository.addConnector(connector); + + repository.removeConnector("connector-1"); + + verify(provider).delete("connector-1"); + } + + @Test + public void testRemoveConnectorProviderThrowsException() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final Connector mockConnector = mock(Connector.class); + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + when(connector.getConnector()).thenReturn(mockConnector); + repository.addConnector(connector); + + doThrow(new ConnectorConfigurationProviderException("Delete failed")).when(provider).delete("connector-1"); + + assertThrows(ConnectorConfigurationProviderException.class, () -> repository.removeConnector("connector-1")); + } + + @Test + public void testUpdateConnectorSavesNameToProvider() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final ConnectorNode connector = createConnectorNodeWithEmptyWorkingConfig("connector-1", "Old Name"); + repository.addConnector(connector); + + repository.updateConnector(connector, "New Name"); + + final ArgumentCaptor configCaptor = ArgumentCaptor.forClass(ConnectorWorkingConfiguration.class); + verify(provider).save(eq("connector-1"), configCaptor.capture()); + assertEquals("New Name", configCaptor.getValue().getName()); + + verify(connector).setName("New Name"); + } + + @Test + public void testInheritConfigurationDoesNotCallProvider() throws FlowUpdateException { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + when(provider.load("connector-1")).thenReturn(Optional.empty()); + + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + repository.addConnector(connector); + + // Reset interactions so we can verify that inheritConfiguration itself does not call the provider + reset(provider); + + repository.inheritConfiguration(connector, List.of(), List.of(), null); + + verifyNoInteractions(provider); + } + + @Test + public void testVerifyCreateWithExistingConnectorThrows() { + final StandardConnectorRepository repository = createRepositoryWithProvider(null); + + final ConnectorNode connector = createSimpleConnectorNode("connector-1", "Test Connector"); + repository.addConnector(connector); + + assertThrows(IllegalStateException.class, () -> repository.verifyCreate("connector-1")); + } + + @Test + public void testVerifyCreateWithNewConnectorSucceeds() { + final StandardConnectorRepository repository = createRepositoryWithProvider(null); + + repository.verifyCreate("connector-1"); + } + + @Test + public void testVerifyCreateDelegatesToProvider() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + repository.verifyCreate("connector-1"); + + verify(provider).verifyCreate("connector-1"); + } + + @Test + public void testVerifyCreateProviderRejectsThrows() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + doThrow(new ConnectorConfigurationProviderException("Create not supported")).when(provider).verifyCreate("connector-1"); + + assertThrows(ConnectorConfigurationProviderException.class, () -> repository.verifyCreate("connector-1")); + } + + @Test + public void testVerifyCreateExistingConnectorDoesNotCallProvider() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + when(provider.load("connector-1")).thenReturn(Optional.empty()); + + final ConnectorNode connector = createSimpleConnectorNode("connector-1", "Test Connector"); + repository.addConnector(connector); + + assertThrows(IllegalStateException.class, () -> repository.verifyCreate("connector-1")); + verify(provider, never()).verifyCreate(anyString()); + } + + // --- Asset Lifecycle Tests --- + + @Test + public void testStoreAssetDelegatesToProviderAndReturnsStoredAsset() throws IOException { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(provider, assetManager); + + final Asset storedAsset = mock(Asset.class); + when(storedAsset.getIdentifier()).thenReturn("nifi-uuid-1"); + // After the provider stores the asset locally, the framework retrieves it via getAsset() + when(assetManager.getAsset("nifi-uuid-1")).thenReturn(Optional.of(storedAsset)); + + final InputStream content = new ByteArrayInputStream("test-content".getBytes()); + final Asset result = repository.storeAsset("connector-1", "nifi-uuid-1", "driver.jar", content); + + assertNotNull(result); + assertEquals("nifi-uuid-1", result.getIdentifier()); + // Provider handles both local storage and external upload -- no direct assetManager.saveAsset() call + verify(provider).storeAsset(eq("connector-1"), eq("nifi-uuid-1"), eq("driver.jar"), any(InputStream.class)); + verify(assetManager, never()).saveAsset(any(), any(), any(), any()); + } + + @Test + public void testStoreAssetWrapsProviderFailureAsIOException() throws IOException { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(provider, assetManager); + + doThrow(new RuntimeException("Provider upload failed")) + .when(provider).storeAsset(eq("connector-1"), eq("nifi-uuid-1"), eq("driver.jar"), any(InputStream.class)); + + final InputStream content = new ByteArrayInputStream("test-content".getBytes()); + assertThrows(IOException.class, () -> repository.storeAsset("connector-1", "nifi-uuid-1", "driver.jar", content)); + + // Rollback of local storage is the provider's responsibility; the framework does not call deleteAsset + verify(assetManager, never()).deleteAsset(anyString()); + } + + @Test + public void testStoreAssetWithoutProviderDelegatesToAssetManagerOnly() throws IOException { + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(null, assetManager); + + final Asset localAsset = mock(Asset.class); + when(localAsset.getIdentifier()).thenReturn("nifi-uuid-1"); + when(assetManager.saveAsset(eq("connector-1"), eq("nifi-uuid-1"), eq("driver.jar"), any(InputStream.class))) + .thenReturn(localAsset); + + final Asset result = repository.storeAsset("connector-1", "nifi-uuid-1", "driver.jar", + new ByteArrayInputStream("content".getBytes())); + assertNotNull(result); + verify(assetManager).saveAsset(eq("connector-1"), eq("nifi-uuid-1"), eq("driver.jar"), any(InputStream.class)); + } + + @Test + public void testGetAssetDelegatesToAssetManager() { + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(null, assetManager); + + final Asset mockAsset = mock(Asset.class); + when(mockAsset.getIdentifier()).thenReturn("asset-1"); + when(assetManager.getAsset("asset-1")).thenReturn(Optional.of(mockAsset)); + + final Optional result = repository.getAsset("asset-1"); + assertTrue(result.isPresent()); + assertEquals("asset-1", result.get().getIdentifier()); + } + + @Test + public void testGetAssetsDelegatesToAssetManager() { + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(null, assetManager); + + final Asset asset1 = mock(Asset.class); + final Asset asset2 = mock(Asset.class); + when(assetManager.getAssets("connector-1")).thenReturn(List.of(asset1, asset2)); + + final List result = repository.getAssets("connector-1"); + assertEquals(2, result.size()); + } + + @Test + public void testDeleteAssetsDelegatesToProviderByNifiUuid() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(provider, assetManager); + + final Asset asset = mock(Asset.class); + when(asset.getIdentifier()).thenReturn("nifi-uuid-1"); + when(assetManager.getAssets("connector-1")).thenReturn(List.of(asset)); + + repository.deleteAssets("connector-1"); + + // Provider receives NiFi UUID; provider handles external cleanup and local deletion + verify(provider).deleteAsset("connector-1", "nifi-uuid-1"); + verify(assetManager, never()).deleteAsset(anyString()); + } + + @Test + public void testDeleteAssetsWithoutProviderDelegatesToAssetManager() { + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(null, assetManager); + + final Asset asset = mock(Asset.class); + when(asset.getIdentifier()).thenReturn("nifi-uuid-1"); + when(assetManager.getAssets("connector-1")).thenReturn(List.of(asset)); + + repository.deleteAssets("connector-1"); + + verify(assetManager).deleteAsset("nifi-uuid-1"); + } + + @Test + public void testSyncFromProviderAppliesNifiUuidsDirectly() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final MutableConnectorConfigurationContext workingConfigContext = mock(MutableConnectorConfigurationContext.class); + final ConnectorNode connector = createConnectorNodeWithWorkingConfig("connector-1", "Test Connector", workingConfigContext); + repository.addConnector(connector); + + // Provider returns config with NiFi UUIDs (no translation needed in framework) + final VersionedConnectorValueReference assetRef = new VersionedConnectorValueReference(); + assetRef.setValueType("ASSET_REFERENCE"); + assetRef.setAssetIds(Set.of("nifi-uuid-from-provider")); + final VersionedConfigurationStep step = createVersionedStep("step1", Map.of("driver", assetRef)); + + final ConnectorWorkingConfiguration config = new ConnectorWorkingConfiguration(); + config.setName("Test Connector"); + config.setWorkingFlowConfiguration(List.of(step)); + when(provider.load("connector-1")).thenReturn(Optional.of(config)); + + repository.getConnector("connector-1"); + + // Working config is updated with NiFi UUIDs as-is -- no translation in the repository + verify(workingConfigContext).replaceProperties(eq("step1"), any(StepConfiguration.class)); + } + + @Test + public void testSyncAssetsFromProviderCallsSyncAssetsThenReloads() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final StandardConnectorRepository repository = createRepositoryWithProvider(provider); + + final MutableConnectorConfigurationContext workingConfigContext = mock(MutableConnectorConfigurationContext.class); + final ConnectorNode connector = createConnectorNodeWithWorkingConfig("connector-1", "Test Connector", workingConfigContext); + repository.addConnector(connector); + + final ConnectorWorkingConfiguration config = new ConnectorWorkingConfiguration(); + config.setName("Test Connector"); + config.setWorkingFlowConfiguration(List.of()); + when(provider.load("connector-1")).thenReturn(Optional.of(config)); + + repository.syncAssetsFromProvider(connector); + + // Step 1: provider.syncAssets() called + verify(provider).syncAssets("connector-1"); + // Step 2: provider.load() called to reload updated config (may also have been called during addConnector) + verify(provider, org.mockito.Mockito.atLeastOnce()).load("connector-1"); + } + + @Test + public void testSyncAssetsFromProviderNoOpWithoutProvider() { + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(null, assetManager); + + final ConnectorNode connector = createSimpleConnectorNode("connector-1", "Test"); + repository.addConnector(connector); + + repository.syncAssetsFromProvider(connector); + + verifyNoInteractions(assetManager); + } + + @Test + public void testCleanUpAssetsCallsProviderDeleteForUnreferencedAssets() { + final ConnectorConfigurationProvider provider = mock(ConnectorConfigurationProvider.class); + final AssetManager assetManager = mock(AssetManager.class); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(provider, assetManager); + + final String connectorId = "connector-1"; + final String referencedAssetId = "referenced-uuid"; + final String unreferencedAssetId = "unreferenced-uuid"; + + final Asset referencedAsset = mock(Asset.class); + when(referencedAsset.getIdentifier()).thenReturn(referencedAssetId); + when(referencedAsset.getName()).thenReturn("referenced.jar"); + + final Asset unreferencedAsset = mock(Asset.class); + when(unreferencedAsset.getIdentifier()).thenReturn(unreferencedAssetId); + when(unreferencedAsset.getName()).thenReturn("unreferenced.jar"); + + final MutableConnectorConfigurationContext activeConfigContext = mock(MutableConnectorConfigurationContext.class); + final ConnectorConfiguration activeConfig = new ConnectorConfiguration(Set.of( + new NamedStepConfiguration("step1", new StepConfiguration( + Map.of("prop", new AssetReference(Set.of(referencedAssetId))))) + )); + when(activeConfigContext.toConnectorConfiguration()).thenReturn(activeConfig); + final FrameworkFlowContext activeFlowContext = mock(FrameworkFlowContext.class); + when(activeFlowContext.getConfigurationContext()).thenReturn(activeConfigContext); + + final MutableConnectorConfigurationContext workingConfigContext = mock(MutableConnectorConfigurationContext.class); + final ConnectorConfiguration workingConfig = new ConnectorConfiguration(Set.of()); + when(workingConfigContext.toConnectorConfiguration()).thenReturn(workingConfig); + final FrameworkFlowContext workingFlowContext = mock(FrameworkFlowContext.class); + when(workingFlowContext.getConfigurationContext()).thenReturn(workingConfigContext); + + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn(connectorId); + when(connector.getActiveFlowContext()).thenReturn(activeFlowContext); + when(connector.getWorkingFlowContext()).thenReturn(workingFlowContext); + + when(assetManager.getAssets(connectorId)).thenReturn(List.of(referencedAsset, unreferencedAsset)); + when(provider.load(connectorId)).thenReturn(Optional.empty()); + repository.addConnector(connector); + + repository.discardWorkingConfiguration(connector); + + // Provider.deleteAsset called with NiFi UUID for unreferenced asset + verify(provider).deleteAsset(connectorId, unreferencedAssetId); + // Referenced asset is not deleted + verify(provider, never()).deleteAsset(connectorId, referencedAssetId); + // AssetManager.deleteAsset NOT called directly since provider handles local deletion + verify(assetManager, never()).deleteAsset(anyString()); + } + + @Test + public void testInheritConfigurationFailureCallsAbortUpdateAndMarkInvalid() throws FlowUpdateException { + final StandardConnectorRepository repository = createRepositoryWithProvider(null); + + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + doThrow(new FlowUpdateException("Simulated failure")) + .when(connector).inheritConfiguration(any(), any(), any()); + + repository.addConnector(connector); + + assertThrows(FlowUpdateException.class, () -> repository.inheritConfiguration(connector, List.of(), List.of(), null)); + + verify(connector).abortUpdate(any(FlowUpdateException.class)); + verify(connector).markInvalid(eq("Flow Update Failure"), eq("The flow could not be updated: Simulated failure")); + } + + @Test + public void testApplyUpdateFailureCallsAbortUpdateButNotMarkInvalid() throws FlowUpdateException { + final AssetManager assetManager = mock(AssetManager.class); + when(assetManager.getAssets("connector-1")).thenReturn(List.of()); + final StandardConnectorRepository repository = createRepositoryWithProviderAndAssetManager(null, assetManager); + + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn("connector-1"); + when(connector.getDesiredState()).thenReturn(ConnectorState.STOPPED); + doThrow(new FlowUpdateException("Simulated failure")) + .when(connector).prepareForUpdate(); + + repository.addConnector(connector); + + repository.applyUpdate(connector, mock(ConnectorUpdateContext.class)); + + verify(connector, timeout(5000)).abortUpdate(any(FlowUpdateException.class)); + verify(connector, never()).markInvalid(anyString(), anyString()); + } + + // --- Helper Methods --- + + private StandardConnectorRepository createRepositoryWithProviderAndAssetManager( + final ConnectorConfigurationProvider provider, final AssetManager assetManager) { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + final ConnectorRepositoryInitializationContext initContext = mock(ConnectorRepositoryInitializationContext.class); + when(initContext.getExtensionManager()).thenReturn(mock(ExtensionManager.class)); + when(initContext.getAssetManager()).thenReturn(assetManager); + when(initContext.getConnectorConfigurationProvider()).thenReturn(provider); + repository.initialize(initContext); + return repository; + } + + private StandardConnectorRepository createRepositoryWithProvider(final ConnectorConfigurationProvider provider) { + final StandardConnectorRepository repository = new StandardConnectorRepository(); + final ConnectorRepositoryInitializationContext initContext = mock(ConnectorRepositoryInitializationContext.class); + when(initContext.getExtensionManager()).thenReturn(mock(ExtensionManager.class)); + when(initContext.getAssetManager()).thenReturn(mock(AssetManager.class)); + when(initContext.getConnectorConfigurationProvider()).thenReturn(provider); + repository.initialize(initContext); + return repository; + } + + private ConnectorNode createSimpleConnectorNode(final String id, final String name) { + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn(id); + when(connector.getName()).thenReturn(name); + return connector; + } + + private ConnectorNode createConnectorNodeWithWorkingConfig(final String id, final String name, final MutableConnectorConfigurationContext workingConfigContext) { + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn(id); + when(connector.getName()).thenReturn(name); + + final FrameworkFlowContext workingFlowContext = mock(FrameworkFlowContext.class); + when(workingFlowContext.getConfigurationContext()).thenReturn(workingConfigContext); + when(connector.getWorkingFlowContext()).thenReturn(workingFlowContext); + + return connector; + } + + private ConnectorNode createConnectorNodeWithEmptyWorkingConfig(final String id, final String name) { + final ConnectorNode connector = mock(ConnectorNode.class); + when(connector.getIdentifier()).thenReturn(id); + when(connector.getName()).thenReturn(name); + + final MutableConnectorConfigurationContext workingConfigContext = mock(MutableConnectorConfigurationContext.class); + final ConnectorConfiguration emptyConfig = new ConnectorConfiguration(Set.of()); + when(workingConfigContext.toConnectorConfiguration()).thenReturn(emptyConfig); + + final FrameworkFlowContext workingFlowContext = mock(FrameworkFlowContext.class); + when(workingFlowContext.getConfigurationContext()).thenReturn(workingConfigContext); + when(connector.getWorkingFlowContext()).thenReturn(workingFlowContext); + + final FrameworkFlowContext activeFlowContext = mock(FrameworkFlowContext.class); + final MutableConnectorConfigurationContext activeConfigContext = mock(MutableConnectorConfigurationContext.class); + when(activeConfigContext.toConnectorConfiguration()).thenReturn(emptyConfig); + when(activeFlowContext.getConfigurationContext()).thenReturn(activeConfigContext); + when(connector.getActiveFlowContext()).thenReturn(activeFlowContext); + + return connector; + } + + private VersionedConfigurationStep createVersionedStep(final String name, final Map properties) { + final VersionedConfigurationStep step = new VersionedConfigurationStep(); + step.setName(name); + step.setProperties(new HashMap<>(properties)); + return step; + } + + private VersionedConnectorValueReference createStringLiteralRef(final String value) { + final VersionedConnectorValueReference ref = new VersionedConnectorValueReference(); + ref.setValueType("STRING_LITERAL"); + ref.setValue(value); + return ref; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/facades/standalone/TestStandaloneProcessGroupLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/facades/standalone/TestStandaloneProcessGroupLifecycle.java new file mode 100644 index 000000000000..6aa4670aa5d5 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/facades/standalone/TestStandaloneProcessGroupLifecycle.java @@ -0,0 +1,292 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.facades.standalone; + +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; +import org.apache.nifi.controller.ControllerService; +import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.controller.service.ControllerServiceNode; +import org.apache.nifi.controller.service.ControllerServiceProvider; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.parameter.ParameterLookup; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Function; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class TestStandaloneProcessGroupLifecycle { + + private static final String SERVICE_ID = "service-1"; + private static final String SERVICE_ID_2 = "service-2"; + + private static final AllowableValue VALUE_A = new AllowableValue("A", "Value A"); + private static final AllowableValue VALUE_B = new AllowableValue("B", "Value B"); + + private static final PropertyDescriptor MODE_PROPERTY = new PropertyDescriptor.Builder() + .name("mode") + .displayName("Mode") + .allowableValues(VALUE_A, VALUE_B) + .defaultValue(VALUE_A.getValue()) + .required(true) + .build(); + + private static final PropertyDescriptor SERVICE_PROPERTY_NO_DEPENDENCY = new PropertyDescriptor.Builder() + .name("service-no-dep") + .displayName("Service (No Dependency)") + .identifiesControllerService(ControllerService.class) + .build(); + + private static final PropertyDescriptor SERVICE_PROPERTY_WITH_DEPENDENCY = new PropertyDescriptor.Builder() + .name("service-with-dep") + .displayName("Service (With Dependency)") + .identifiesControllerService(ControllerService.class) + .dependsOn(MODE_PROPERTY, VALUE_B) + .build(); + + @Mock + private ProcessGroup processGroup; + + @Mock + private ControllerServiceProvider controllerServiceProvider; + + @Mock + private StatelessGroupLifecycle statelessGroupLifecycle; + + private StandaloneProcessGroupLifecycle lifecycle; + + @BeforeEach + void setUp() { + lifecycle = new StandaloneProcessGroupLifecycle(processGroup, controllerServiceProvider, statelessGroupLifecycle, id -> null); + lenient().when(processGroup.getProcessGroups()).thenReturn(Collections.emptySet()); + } + + @Test + void testFindReferencedServicesIncludesServiceWithNoDependency() { + final ControllerServiceNode serviceNode = mock(ControllerServiceNode.class); + when(serviceNode.getRequiredControllerServices()).thenReturn(Collections.emptyList()); + + final ProcessorNode processor = createProcessorNode( + List.of(SERVICE_PROPERTY_NO_DEPENDENCY), + Map.of(SERVICE_PROPERTY_NO_DEPENDENCY, SERVICE_ID) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + when(controllerServiceProvider.getControllerServiceNode(SERVICE_ID)).thenReturn(serviceNode); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.contains(serviceNode), "Service with no property dependency should be included"); + assertEquals(1, result.size()); + } + + @Test + void testFindReferencedServicesExcludesServiceWhenDependencyNotSatisfied() { + final ProcessorNode processor = createProcessorNode( + List.of(MODE_PROPERTY, SERVICE_PROPERTY_WITH_DEPENDENCY), + Map.of(MODE_PROPERTY, VALUE_A.getValue(), SERVICE_PROPERTY_WITH_DEPENDENCY, SERVICE_ID) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.isEmpty(), "Service should not be included when property dependency is not satisfied"); + } + + @Test + void testFindReferencedServicesIncludesServiceWhenDependencySatisfied() { + final ControllerServiceNode serviceNode = mock(ControllerServiceNode.class); + when(serviceNode.getRequiredControllerServices()).thenReturn(Collections.emptyList()); + + final ProcessorNode processor = createProcessorNode( + List.of(MODE_PROPERTY, SERVICE_PROPERTY_WITH_DEPENDENCY), + Map.of(MODE_PROPERTY, VALUE_B.getValue(), SERVICE_PROPERTY_WITH_DEPENDENCY, SERVICE_ID) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + when(controllerServiceProvider.getControllerServiceNode(SERVICE_ID)).thenReturn(serviceNode); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.contains(serviceNode), "Service should be included when property dependency is satisfied"); + assertEquals(1, result.size()); + } + + @Test + void testFindReferencedServicesUsesDefaultValueForDependencyCheck() { + // A property that depends on MODE_PROPERTY having VALUE_A (which is the default) + final PropertyDescriptor servicePropertyDependsOnDefault = new PropertyDescriptor.Builder() + .name("service-dep-default") + .displayName("Service (Depends on Default)") + .identifiesControllerService(ControllerService.class) + .dependsOn(MODE_PROPERTY, VALUE_A) + .build(); + + final ControllerServiceNode serviceNode = mock(ControllerServiceNode.class); + when(serviceNode.getRequiredControllerServices()).thenReturn(Collections.emptyList()); + + // Mode property not explicitly set; the default value "A" should satisfy the dependency + final ProcessorNode processor = createProcessorNode( + List.of(MODE_PROPERTY, servicePropertyDependsOnDefault), + Map.of(servicePropertyDependsOnDefault, SERVICE_ID) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + when(controllerServiceProvider.getControllerServiceNode(SERVICE_ID)).thenReturn(serviceNode); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.contains(serviceNode), "Service should be included when dependency property uses default value that satisfies dependency"); + assertEquals(1, result.size()); + } + + @Test + void testFindReferencedServicesTransitiveServiceExcludedWhenDependencyNotSatisfied() { + // Service 1 is directly referenced by the processor (no dependency on service property) + final ControllerServiceNode serviceNode1 = mock(ControllerServiceNode.class); + // getRequiredControllerServices() already filters based on dependencies, so it returns empty + when(serviceNode1.getRequiredControllerServices()).thenReturn(Collections.emptyList()); + + final ProcessorNode processor = createProcessorNode( + List.of(SERVICE_PROPERTY_NO_DEPENDENCY), + Map.of(SERVICE_PROPERTY_NO_DEPENDENCY, SERVICE_ID) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + when(controllerServiceProvider.getControllerServiceNode(SERVICE_ID)).thenReturn(serviceNode1); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.contains(serviceNode1), "Directly referenced service should be included"); + assertEquals(1, result.size(), "Only the directly referenced service should be in the result"); + } + + @Test + void testFindReferencedServicesTransitiveServiceIncludedWhenDependencySatisfied() { + final ControllerServiceNode serviceNode1 = mock(ControllerServiceNode.class); + final ControllerServiceNode serviceNode2 = mock(ControllerServiceNode.class); + // Service 1 requires Service 2 (dependency is satisfied) + when(serviceNode1.getRequiredControllerServices()).thenReturn(List.of(serviceNode2)); + when(serviceNode2.getRequiredControllerServices()).thenReturn(Collections.emptyList()); + + final ProcessorNode processor = createProcessorNode( + List.of(SERVICE_PROPERTY_NO_DEPENDENCY), + Map.of(SERVICE_PROPERTY_NO_DEPENDENCY, SERVICE_ID) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + when(controllerServiceProvider.getControllerServiceNode(SERVICE_ID)).thenReturn(serviceNode1); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.contains(serviceNode1), "Directly referenced service should be included"); + assertTrue(result.contains(serviceNode2), "Transitively required service should be included"); + assertEquals(2, result.size()); + } + + @Test + void testFindReferencedServicesMultiplePropertiesMixedDependencies() { + final ControllerServiceNode serviceNode1 = mock(ControllerServiceNode.class); + when(serviceNode1.getRequiredControllerServices()).thenReturn(Collections.emptyList()); + final ControllerServiceNode serviceNode2 = mock(ControllerServiceNode.class); + + // Processor with two service properties: one with no dependency (service1) and one with unsatisfied dependency (service2) + // Mode is "A", which does NOT satisfy the dependency on VALUE_B for SERVICE_PROPERTY_WITH_DEPENDENCY + final ProcessorNode processor = createProcessorNode( + List.of(MODE_PROPERTY, SERVICE_PROPERTY_NO_DEPENDENCY, SERVICE_PROPERTY_WITH_DEPENDENCY), + Map.of(MODE_PROPERTY, VALUE_A.getValue(), SERVICE_PROPERTY_NO_DEPENDENCY, SERVICE_ID, SERVICE_PROPERTY_WITH_DEPENDENCY, SERVICE_ID_2) + ); + + when(processGroup.getProcessors()).thenReturn(List.of(processor)); + when(controllerServiceProvider.getControllerServiceNode(SERVICE_ID)).thenReturn(serviceNode1); + + final Set result = lifecycle.findReferencedServices(false); + + assertTrue(result.contains(serviceNode1), "Service from property with no dependency should be included"); + assertFalse(result.contains(serviceNode2), "Service from property with unsatisfied dependency should not be included"); + assertEquals(1, result.size()); + } + + /** + * Creates a mock ProcessorNode with the given property descriptors and effective property values. + * The mock is configured with a ValidationContext whose isDependencySatisfied calls the real default + * implementation, using property values derived from the provided effective values. + */ + private ProcessorNode createProcessorNode(final List descriptors, + final Map effectiveValues) { + final ProcessorNode processor = mock(ProcessorNode.class); + when(processor.getPropertyDescriptors()).thenReturn(descriptors); + + for (final PropertyDescriptor descriptor : descriptors) { + lenient().when(processor.getPropertyDescriptor(descriptor.getName())).thenReturn(descriptor); + lenient().when(processor.getEffectivePropertyValue(descriptor)).thenReturn(effectiveValues.get(descriptor)); + } + + // Build the effective property value map for creating the validation context + final Map effectivePropertyValues = new LinkedHashMap<>(); + for (final PropertyDescriptor descriptor : descriptors) { + final String value = effectiveValues.get(descriptor); + if (value != null) { + effectivePropertyValues.put(descriptor, value); + } + } + when(processor.getEffectivePropertyValues()).thenReturn(effectivePropertyValues); + + // Set up a ValidationContext that delegates isDependencySatisfied to the real default implementation + final ValidationContext validationContext = mock(ValidationContext.class); + when(validationContext.isDependencySatisfied(any(PropertyDescriptor.class), any(Function.class))).thenCallRealMethod(); + + // Mock getProperty() so the real isDependencySatisfied default method can resolve property values + for (final PropertyDescriptor descriptor : descriptors) { + final String value = effectiveValues.get(descriptor); + if (value != null) { + final PropertyValue propertyValue = mock(PropertyValue.class); + lenient().when(propertyValue.getValue()).thenReturn(value); + lenient().when(validationContext.getProperty(descriptor)).thenReturn(propertyValue); + } + } + + lenient().when(processor.getAnnotationData()).thenReturn(null); + lenient().when(processor.getParameterLookup()).thenReturn(ParameterLookup.EMPTY); + when(processor.createValidationContext(any(Map.class), any(), any(ParameterLookup.class), anyBoolean())).thenReturn(validationContext); + + return processor; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/CreateDummyFlowFile.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/CreateDummyFlowFile.java new file mode 100644 index 000000000000..98ead292e1b6 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/CreateDummyFlowFile.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.services.CounterService; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Set; + +@InputRequirement(Requirement.INPUT_FORBIDDEN) +public class CreateDummyFlowFile extends AbstractProcessor { + + static final PropertyDescriptor TEXT = new PropertyDescriptor.Builder() + .name("Text") + .description("The text to write to the FlowFile content") + .required(false) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + static final PropertyDescriptor COUNT_SERVICE = new PropertyDescriptor.Builder() + .name("Counter Service") + .description("The Counter Service to use for incrementing a counter each time a FlowFile is created") + .required(false) + .identifiesControllerService(CounterService.class) + .build(); + + private static final Relationship REL_SUCCESS = new Relationship.Builder() + .name("success") + .description("All FlowFiles are routed to this relationship") + .build(); + + @Override + protected List getSupportedPropertyDescriptors() { + return List.of(TEXT); + } + + @Override + public Set getRelationships() { + return Set.of(REL_SUCCESS); + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + FlowFile flowFile = session.create(); + final String text = context.getProperty(TEXT).getValue(); + if (text != null) { + flowFile = session.write(flowFile, out -> out.write(text.getBytes(StandardCharsets.UTF_8))); + } + + final CounterService service = context.getProperty(COUNT_SERVICE).asControllerService(CounterService.class); + if (service != null) { + final long count = service.increment(); + flowFile = session.putAttribute(flowFile, "counter.value", String.valueOf(count)); + } + + session.transfer(flowFile, REL_SUCCESS); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/DuplicateFlowFile.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/DuplicateFlowFile.java new file mode 100644 index 000000000000..34c438ba61a3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/DuplicateFlowFile.java @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; + +@Tags({"test", "duplicate", "connector"}) +@InputRequirement(Requirement.INPUT_REQUIRED) +@CapabilityDescription("Duplicates incoming FlowFiles to numbered relationships based on the configured number of duplicates. " + + "Each duplicate is sent to a relationship numbered 1, 2, 3, etc., up to the configured number.") +public class DuplicateFlowFile extends AbstractProcessor { + + public static final PropertyDescriptor NUM_DUPLICATES = new PropertyDescriptor.Builder() + .name("Number of Duplicates") + .description("Specifies how many duplicates of each incoming FlowFile will be created") + .required(true) + .defaultValue("3") + .addValidator(StandardValidators.createLongValidator(1L, 10L, true)) + .build(); + + private static final List PROPERTY_DESCRIPTORS = List.of(NUM_DUPLICATES); + + private final AtomicReference> relationshipsRef = new AtomicReference<>(); + private final AtomicReference> sortedRelationshipsRef = new AtomicReference<>(); + + @Override + protected List getSupportedPropertyDescriptors() { + return PROPERTY_DESCRIPTORS; + } + + @Override + public Set getRelationships() { + final Set relationships = relationshipsRef.get(); + return relationships == null ? Set.of() : relationships; + } + + @Override + public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) { + if (NUM_DUPLICATES.equals(descriptor)) { + updateRelationships(newValue); + } + } + + private void updateRelationships(final String numDuplicatesValue) { + if (numDuplicatesValue == null) { + relationshipsRef.set(Set.of()); + sortedRelationshipsRef.set(List.of()); + return; + } + + try { + final int numDuplicates = Integer.parseInt(numDuplicatesValue); + final Set relationships = new HashSet<>(); + final List sortedRelationships = new ArrayList<>(); + + for (int i = 1; i <= numDuplicates; i++) { + final Relationship relationship = new Relationship.Builder() + .name(String.valueOf(i)) + .description("Relationship for duplicate " + i) + .build(); + relationships.add(relationship); + sortedRelationships.add(relationship); + } + + relationshipsRef.set(relationships); + sortedRelationshipsRef.set(sortedRelationships); + } catch (final NumberFormatException e) { + relationshipsRef.set(Set.of()); + sortedRelationshipsRef.set(List.of()); + } + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + FlowFile flowFile = session.get(); + if (flowFile == null) { + return; + } + + final List sortedRelationships = sortedRelationshipsRef.get(); + + if (sortedRelationships == null || sortedRelationships.isEmpty()) { + session.rollback(); + return; + } + + // Create duplicates and send to numbered relationships + for (final Relationship relationship : sortedRelationships) { + final FlowFile duplicate = session.clone(flowFile); + session.transfer(duplicate, relationship); + } + + // Remove the original FlowFile + session.remove(flowFile); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/ExposeFileValues.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/ExposeFileValues.java new file mode 100644 index 000000000000..154e3d2b5fe5 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/ExposeFileValues.java @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.components.ComponentState; +import org.apache.nifi.components.connector.components.ConnectorMethod; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.List; + +public class ExposeFileValues extends AbstractProcessor { + + static final PropertyDescriptor FILE = new PropertyDescriptor.Builder() + .name("File") + .description("The name of the file whose contents will be exposed via a ConnectorMethod.") + .required(true) + .addValidator(StandardValidators.FILE_EXISTS_VALIDATOR) + .build(); + + private volatile String filename; + + @Override + protected List getSupportedPropertyDescriptors() { + return List.of(FILE); + } + + @Override + public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) { + if (descriptor.equals(FILE)) { + this.filename = newValue; + } + } + + @ConnectorMethod( + name = "getFileValues", + description = "Reads all lines from the configured file and returns them as a List of String.", + allowedStates = ComponentState.STOPPED + ) + public List getFileValues() throws IOException { + final File file = new File(filename); + return Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); + } + + @Override + public void onTrigger(final ProcessContext processContext, final ProcessSession processSession) throws ProcessException { + + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/LogFlowFileContents.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/LogFlowFileContents.java new file mode 100644 index 000000000000..795387601924 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/LogFlowFileContents.java @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; + +import java.util.Set; + +public class LogFlowFileContents extends AbstractProcessor { + + public static final Relationship REL_SUCCESS = new Relationship.Builder() + .name("success") + .description("FlowFiles that have been successfully overwritten") + .build(); + + private static final Set RELATIONSHIPS = Set.of(REL_SUCCESS); + + @Override + public Set getRelationships() { + return RELATIONSHIPS; + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + FlowFile flowFile = session.get(); + if (flowFile == null) { + return; + } + + session.read(flowFile, in -> { + final byte[] contents = in.readAllBytes(); + final String contentStr = new String(contents); + getLogger().info("FlowFile Content:\n{}", contentStr); + }); + + session.transfer(flowFile, REL_SUCCESS); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/OnPropertyModifiedTracker.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/OnPropertyModifiedTracker.java new file mode 100644 index 000000000000..ad8de1eda663 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/OnPropertyModifiedTracker.java @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.annotation.lifecycle.OnConfigurationRestored; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Set; + +/** + * A test processor that tracks calls to onPropertyModified when the Configured Number property changes. + * This is used to verify that onPropertyModified is called correctly when a Connector's applyUpdate + * changes a parameter value. + */ +public class OnPropertyModifiedTracker extends AbstractProcessor { + + static final PropertyDescriptor CONFIGURED_NUMBER = new PropertyDescriptor.Builder() + .name("Configured Number") + .displayName("Configured Number") + .description("A number property that is tracked for changes via onPropertyModified") + .required(false) + .addValidator(StandardValidators.INTEGER_VALIDATOR) + .build(); + + static final Relationship REL_SUCCESS = new Relationship.Builder() + .name("success") + .description("All FlowFiles are routed here") + .build(); + + private volatile boolean configurationRestored = false; + private final List propertyChanges = Collections.synchronizedList(new ArrayList<>()); + + @Override + protected List getSupportedPropertyDescriptors() { + return List.of(CONFIGURED_NUMBER); + } + + @Override + public Set getRelationships() { + return Set.of(REL_SUCCESS); + } + + @OnConfigurationRestored + public void onConfigurationRestored() { + this.configurationRestored = true; + } + + @Override + public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) { + if (!CONFIGURED_NUMBER.getName().equals(descriptor.getName())) { + return; + } + + if (!configurationRestored) { + return; + } + + if (oldValue == null) { + getLogger().info("Property [{}] initialized to [{}]", descriptor.getName(), newValue); + return; + } + + getLogger().info("Property [{}] changed from [{}] to [{}]", descriptor.getName(), oldValue, newValue); + propertyChanges.add(new PropertyChange(oldValue, newValue)); + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + FlowFile flowFile = session.get(); + if (flowFile == null) { + return; + } + + session.transfer(flowFile, REL_SUCCESS); + } + + public List getPropertyChanges() { + return new ArrayList<>(propertyChanges); + } + + public int getPropertyChangeCount() { + return propertyChanges.size(); + } + + public void clearPropertyChanges() { + propertyChanges.clear(); + } + + public record PropertyChange(String oldValue, String newValue) { + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/OverwriteFlowFile.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/OverwriteFlowFile.java new file mode 100644 index 000000000000..83632081e46e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/OverwriteFlowFile.java @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.expression.ExpressionLanguageScope; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.OutputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; + +import java.io.IOException; +import java.io.OutputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Set; + +@Tags({"test", "overwrite", "content", "connector"}) +@InputRequirement(Requirement.INPUT_REQUIRED) +@CapabilityDescription("Overwrites the content of incoming FlowFiles with a configurable string value and transfers them to the success relationship.") +public class OverwriteFlowFile extends AbstractProcessor { + + public static final PropertyDescriptor CONTENT = new PropertyDescriptor.Builder() + .name("Content") + .description("The content to write to the FlowFile") + .required(true) + .defaultValue("Hello World") + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) + .build(); + + private static final List PROPERTY_DESCRIPTORS = List.of(CONTENT); + + public static final Relationship REL_SUCCESS = new Relationship.Builder() + .name("success") + .description("FlowFiles that have been successfully overwritten") + .build(); + + private static final Set RELATIONSHIPS = Set.of(REL_SUCCESS); + + @Override + protected List getSupportedPropertyDescriptors() { + return PROPERTY_DESCRIPTORS; + } + + @Override + public Set getRelationships() { + return RELATIONSHIPS; + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + FlowFile flowFile = session.get(); + if (flowFile == null) { + return; + } + + final String content = context.getProperty(CONTENT).evaluateAttributeExpressions(flowFile).getValue(); + + flowFile = session.write(flowFile, new OutputStreamCallback() { + @Override + public void process(final OutputStream out) throws IOException { + out.write(content.getBytes(StandardCharsets.UTF_8)); + } + }); + + session.transfer(flowFile, REL_SUCCESS); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/Sleep.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/Sleep.java new file mode 100644 index 000000000000..f936ece3f2f8 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/Sleep.java @@ -0,0 +1,151 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.expression.ExpressionLanguageScope; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; + +@Tags({"test", "sleep", "delay", "timing"}) +@CapabilityDescription("Test processor that introduces configurable delays during different lifecycle events. " + + "Can sleep during OnScheduled, OnStopped, and/or onTrigger methods.") +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@SideEffectFree +public class Sleep extends AbstractProcessor { + + static final PropertyDescriptor SLEEP_DURATION = new PropertyDescriptor.Builder() + .name("Sleep Duration") + .displayName("Sleep Duration") + .description("Length of time to sleep when enabled.") + .required(true) + .defaultValue("5 sec") + .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR) + .expressionLanguageSupported(ExpressionLanguageScope.NONE) + .build(); + + static final PropertyDescriptor SLEEP_ON_SCHEDULED = new PropertyDescriptor.Builder() + .name("Sleep On Scheduled") + .displayName("Sleep On Scheduled") + .description("Sleep during OnScheduled when enabled.") + .required(true) + .allowableValues("true", "false") + .defaultValue("false") + .addValidator(StandardValidators.BOOLEAN_VALIDATOR) + .build(); + + static final PropertyDescriptor SLEEP_ON_TRIGGER = new PropertyDescriptor.Builder() + .name("Sleep On Trigger") + .displayName("Sleep On Trigger") + .description("Sleep during onTrigger when enabled.") + .required(true) + .allowableValues("true", "false") + .defaultValue("false") + .addValidator(StandardValidators.BOOLEAN_VALIDATOR) + .build(); + + static final PropertyDescriptor SLEEP_ON_STOPPED = new PropertyDescriptor.Builder() + .name("Sleep On Stopped") + .displayName("Sleep On Stopped") + .description("Sleep during OnStopped when enabled.") + .required(true) + .allowableValues("true", "false") + .defaultValue("false") + .addValidator(StandardValidators.BOOLEAN_VALIDATOR) + .build(); + + static final Relationship REL_SUCCESS = new Relationship.Builder() + .name("success") + .description("All FlowFiles are routed to success after optional sleeping.") + .build(); + + private static final List DESCRIPTORS = List.of( + SLEEP_DURATION, + SLEEP_ON_SCHEDULED, + SLEEP_ON_TRIGGER, + SLEEP_ON_STOPPED + ); + + private static final Set RELATIONSHIPS = Set.of(REL_SUCCESS); + + @Override + public Set getRelationships() { + return RELATIONSHIPS; + } + + @Override + protected List getSupportedPropertyDescriptors() { + return DESCRIPTORS; + } + + @OnScheduled + public void onScheduled(final ProcessContext context) { + if (context.getProperty(SLEEP_ON_SCHEDULED).asBoolean()) { + sleep(context); + } + } + + @OnStopped + public void onStopped(final ProcessContext context) { + if (context.getProperty(SLEEP_ON_STOPPED).asBoolean()) { + sleep(context); + } + } + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) { + if (context.getProperty(SLEEP_ON_TRIGGER).asBoolean()) { + sleep(context); + } + + FlowFile flowFile = session.get(); + if (flowFile == null) { + return; + } + + session.transfer(flowFile, REL_SUCCESS); + } + + private void sleep(final ProcessContext context) { + final long durationMillis = context.getProperty(SLEEP_DURATION).asTimePeriod(TimeUnit.MILLISECONDS); + final ComponentLog logger = getLogger(); + try { + if (durationMillis > 0) { + Thread.sleep(durationMillis); + } + } catch (final InterruptedException interruptedException) { + Thread.currentThread().interrupt(); + logger.warn("Sleep processor interrupted while sleeping for {} ms", durationMillis); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/TerminateFlowFile.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/TerminateFlowFile.java new file mode 100644 index 000000000000..b507b8071dc5 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/processors/TerminateFlowFile.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.processors; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.exception.ProcessException; + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +public class TerminateFlowFile extends AbstractProcessor { + + @Override + public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + FlowFile flowFile = session.get(); + if (flowFile != null) { + session.remove(flowFile); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/services/CounterService.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/services/CounterService.java new file mode 100644 index 000000000000..fe9cebad30f0 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/services/CounterService.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.services; + +import org.apache.nifi.controller.ControllerService; + +public interface CounterService extends ControllerService { + long increment(); + + long getCount(); +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/services/impl/StandardCounterService.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/services/impl/StandardCounterService.java new file mode 100644 index 000000000000..99515f450116 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/components/connector/services/impl/StandardCounterService.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector.services.impl; + +import org.apache.nifi.components.connector.services.CounterService; +import org.apache.nifi.controller.AbstractControllerService; + +import java.util.concurrent.atomic.AtomicLong; + +public class StandardCounterService extends AbstractControllerService implements CounterService { + private final AtomicLong counter = new AtomicLong(); + + @Override + public long increment() { + return counter.incrementAndGet(); + } + + @Override + public long getCount() { + return counter.get(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockStateManagerProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockStateManagerProvider.java new file mode 100644 index 000000000000..e0766c2b3ee0 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockStateManagerProvider.java @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller; + +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateManagerProvider; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.controller.state.StandardStateMap; +import org.mockito.Mockito; + +import java.io.IOException; +import java.util.Collections; +import java.util.Optional; + +import static org.mockito.ArgumentMatchers.any; + +public class MockStateManagerProvider implements StateManagerProvider { + @Override + public StateManager getStateManager(final String componentId) { + final StateManager stateManager = Mockito.mock(StateManager.class); + final StateMap emptyStateMap = new StandardStateMap(Collections.emptyMap(), Optional.empty()); + try { + Mockito.when(stateManager.getState(any(Scope.class))).thenReturn(emptyStateMap); + } catch (IOException e) { + throw new AssertionError(); + } + + return stateManager; + } + + @Override + public StateManager getStateManager(final String componentId, final boolean dropStateKeySupported) { + return getStateManager(componentId); + } + + @Override + public void shutdown() { + } + + @Override + public void enableClusterProvider() { + } + + @Override + public void disableClusterProvider() { + } + + @Override + public boolean isClusterProviderEnabled() { + return false; + } + + @Override + public void onComponentRemoved(final String componentId) { + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockSwapManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockSwapManager.java index 90a0dfbb4a45..3c390be62e43 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockSwapManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/MockSwapManager.java @@ -173,6 +173,11 @@ public void purge() { swappedOut.clear(); } + @Override + public void deleteSwapFile(final String swapLocation) throws IOException { + swappedOut.remove(swapLocation); + } + @Override public String getQueueIdentifier(final String swapLocation) { return null; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestStandardFlowFileQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestStandardFlowFileQueue.java index 8a6469f3e353..e1aac575a0a9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestStandardFlowFileQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestStandardFlowFileQueue.java @@ -17,6 +17,7 @@ package org.apache.nifi.controller; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Connection; import org.apache.nifi.controller.queue.DropFlowFileState; @@ -27,6 +28,8 @@ import org.apache.nifi.controller.queue.StandardFlowFileQueue; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.FlowFileRepository; +import org.apache.nifi.controller.repository.RepositoryRecord; +import org.apache.nifi.controller.repository.RepositoryRecordType; import org.apache.nifi.processor.FlowFileFilter; import org.apache.nifi.processor.FlowFileFilter.FlowFileFilterResult; import org.apache.nifi.provenance.ProvenanceEventRecord; @@ -58,19 +61,20 @@ public class TestStandardFlowFileQueue { private MockSwapManager swapManager = null; private StandardFlowFileQueue queue = null; - private Connection connection = null; private FlowFileRepository flowFileRepo = null; private ProvenanceEventRepository provRepo = null; private ProcessScheduler scheduler = null; - private List provRecords = new ArrayList<>(); + private final List provRecords = new ArrayList<>(); + private final List repoRecords = new ArrayList<>(); @BeforeEach @SuppressWarnings("unchecked") - public void setup() { + public void setup() throws Exception { provRecords.clear(); + repoRecords.clear(); - connection = Mockito.mock(Connection.class); + final Connection connection = Mockito.mock(Connection.class); Mockito.when(connection.getSource()).thenReturn(Mockito.mock(Connectable.class)); Mockito.when(connection.getDestination()).thenReturn(Mockito.mock(Connectable.class)); @@ -89,6 +93,12 @@ public void setup() { return null; }).when(provRepo).registerEvents(Mockito.any(Iterable.class)); + Mockito.doAnswer((Answer) invocation -> { + final Collection records = (Collection) invocation.getArguments()[0]; + repoRecords.addAll(records); + return null; + }).when(flowFileRepo).updateRepository(Mockito.any(Collection.class)); + queue = new StandardFlowFileQueue("id", flowFileRepo, provRepo, scheduler, swapManager, null, 10000, "0 sec", 0L, "0 B"); MockFlowFileRecord.resetIdGenerator(); } @@ -338,24 +348,25 @@ public void testSwapIn() { @Test public void testSwapInWhenThresholdIsLessThanSwapSize() { // create a queue where the swap threshold is less than 10k + // With threshold of 1000, swapRecordPollSize will also be 1000 (min of 10000 and threshold) queue = new StandardFlowFileQueue("id", flowFileRepo, provRepo, scheduler, swapManager, null, 1000, "0 sec", 0L, "0 B"); for (int i = 1; i <= 20000; i++) { queue.put(new MockFlowFileRecord()); } - assertEquals(1, swapManager.swappedOut.size()); + // With swapRecordPollSize = 1000, we expect 19 swap files (19,000 FlowFiles) + assertEquals(19, swapManager.swappedOut.size()); queue.put(new MockFlowFileRecord()); - assertEquals(1, swapManager.swappedOut.size()); + assertEquals(19, swapManager.swappedOut.size()); final Set exp = new HashSet<>(); // At this point there should be: - // 1k FlowFiles in the active queue - // 9,001 FlowFiles in the swap queue - // 10k FlowFiles swapped to disk - - for (int i = 0; i < 999; i++) { // + // 1k flow files in the active queue + // 1 flow file in the swap queue + // 19k flow files swapped to disk (19 swap files with 1k each) + for (int i = 0; i < 999; i++) { final FlowFileRecord flowFile = queue.poll(exp); assertNotNull(flowFile); assertEquals(1, queue.getQueueDiagnostics().getLocalQueuePartitionDiagnostics().getUnacknowledgedQueueSize().getObjectCount()); @@ -373,13 +384,13 @@ public void testSwapInWhenThresholdIsLessThanSwapSize() { assertEquals(0, swapManager.swapInCalledCount); assertEquals(0, queue.getQueueDiagnostics().getLocalQueuePartitionDiagnostics().getActiveQueueSize().getObjectCount()); - assertEquals(1, swapManager.swapOutCalledCount); + assertEquals(19, swapManager.swapOutCalledCount); - assertNotNull(queue.poll(exp)); // this should trigger a swap-in of 10,000 records, and then pull 1 off the top. + assertNotNull(queue.poll(exp)); // this should trigger a swap-in of 1,000 records, and then pull 1 off the top. assertEquals(1, swapManager.swapInCalledCount); - assertEquals(9999, queue.getQueueDiagnostics().getLocalQueuePartitionDiagnostics().getActiveQueueSize().getObjectCount()); + assertEquals(999, queue.getQueueDiagnostics().getLocalQueuePartitionDiagnostics().getActiveQueueSize().getObjectCount()); - assertTrue(swapManager.swappedOut.isEmpty()); + assertEquals(18, swapManager.swappedOut.size()); queue.poll(exp); } @@ -609,4 +620,81 @@ public void testGetMinLastQueueDate() { assertEquals(500, now - queue.getMinLastQueueDate()); } + + @Test + public void testSelectiveDropCreatesDeleteRecords() throws Exception { + for (int i = 0; i < 10; i++) { + queue.put(new MockFlowFileRecord(i)); + } + + final DropFlowFileSummary summary = queue.dropFlowFiles(ff -> ff.getSize() < 5); + assertEquals(5, summary.getDroppedCount()); + + final long deleteRecordCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.DELETE).count(); + assertEquals(5, deleteRecordCount); + + assertEquals(5, provRecords.size()); + for (final ProvenanceEventRecord event : provRecords) { + assertEquals(ProvenanceEventType.DROP, event.getEventType()); + } + + assertEquals(5, queue.size().getObjectCount()); + } + + @Test + public void testSelectiveDropWithSwappedFlowFilesCreatesSwapFileRenamedRecords() throws Exception { + for (int i = 0; i < 20000; i++) { + queue.put(new MockFlowFileRecord(i % 10)); + } + + assertEquals(1, swapManager.swappedOut.size()); + + repoRecords.clear(); + + final DropFlowFileSummary summary = queue.dropFlowFiles(ff -> ff.getSize() < 5); + assertEquals(10000, summary.getDroppedCount()); + + final long deleteRecordCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.DELETE).count(); + assertEquals(10000, deleteRecordCount); + + final long swapFileRenamedCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_RENAMED).count(); + assertEquals(1, swapFileRenamedCount); + + final RepositoryRecord swapRenamedRecord = repoRecords.stream() + .filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_RENAMED) + .findFirst() + .orElseThrow(); + assertNotNull(swapRenamedRecord.getOriginalSwapLocation()); + assertNotNull(swapRenamedRecord.getSwapLocation()); + + assertEquals(10000, queue.size().getObjectCount()); + } + + @Test + public void testSelectiveDropWithAllSwappedFlowFilesCreatesSwapFileDeletedRecords() throws Exception { + for (int i = 0; i < 20000; i++) { + queue.put(new MockFlowFileRecord(1)); + } + + assertEquals(1, swapManager.swappedOut.size()); + + repoRecords.clear(); + + final DropFlowFileSummary summary = queue.dropFlowFiles(ff -> ff.getSize() == 1); + assertEquals(20000, summary.getDroppedCount()); + + final long deleteRecordCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.DELETE).count(); + assertEquals(20000, deleteRecordCount); + + final long swapFileDeletedCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_DELETED).count(); + assertEquals(1, swapFileDeletedCount); + + final RepositoryRecord swapDeletedRecord = repoRecords.stream() + .filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_DELETED) + .findFirst() + .orElseThrow(); + assertNotNull(swapDeletedRecord.getSwapLocation()); + + assertEquals(0, queue.size().getObjectCount()); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/flow/NopConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/flow/NopConnector.java new file mode 100644 index 000000000000..c2dc1ab49a89 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/flow/NopConnector.java @@ -0,0 +1,167 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller.flow; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.ConnectorInitializationContext; +import org.apache.nifi.components.connector.ConnectorValidationContext; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +/** + * A simple no-op Connector implementation for testing purposes. + * This connector does nothing but provides the basic interface implementation + * required for testing StandardFlowManager's connector creation functionality. + */ +public class NopConnector implements Connector { + + private ConnectorInitializationContext context; + private boolean initialized = false; + private boolean started = false; + private boolean configured = false; + + @Override + public void initialize(final ConnectorInitializationContext context) { + this.context = context; + this.initialized = true; + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void start(final FlowContext activeContext) throws FlowUpdateException { + if (!initialized) { + throw new FlowUpdateException("Connector must be initialized before starting"); + } + started = true; + } + + @Override + public void stop(final FlowContext activeContext) throws FlowUpdateException { + started = false; + } + + @Override + public List validate(final FlowContext activeContext, final ConnectorValidationContext validationContext) { + if (!initialized) { + return List.of(new ValidationResult.Builder() + .subject("Initialization") + .valid(false) + .explanation("Connector has not been initialized") + .build()); + } + + return List.of(new ValidationResult.Builder() + .subject("Test Connector") + .valid(true) + .explanation("Test connector is valid") + .build()); + } + + @Override + public List validateConfigurationStep(final ConfigurationStep configurationStep, final ConnectorConfigurationContext connectorConfigurationContext, + final ConnectorValidationContext connectorValidationContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(new ConfigurationStep.Builder() + .name("Test Group") + .description("A test configuration step") + .build()); + } + + @Override + public void onConfigurationStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + + } + + @Override + public void abortUpdate(final FlowContext flowContext, final Throwable throwable) { + + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + if (!initialized) { + throw new FlowUpdateException("Connector must be initialized before configuration"); + } + configured = true; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List verify(final FlowContext flowContext) { + return List.of(); + } + + + // Getters for test assertions + public boolean isInitialized() { + return initialized; + } + + public boolean isStarted() { + return started; + } + + public boolean isConfigured() { + return configured; + } + + public ConnectorInitializationContext getContext() { + return context; + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext, final String filter) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext workingContext) { + return List.of(); + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + return CompletableFuture.completedFuture(null); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/flow/TestStandardFlowManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/flow/TestStandardFlowManager.java new file mode 100644 index 000000000000..473962890fdd --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/flow/TestStandardFlowManager.java @@ -0,0 +1,174 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.controller.flow; + +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorInitializationContext; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorValidationTrigger; +import org.apache.nifi.components.connector.StandardConnectorInitializationContext; +import org.apache.nifi.components.connector.StandardConnectorStateTransition; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.GarbageCollectionLog; +import org.apache.nifi.controller.MockStateManagerProvider; +import org.apache.nifi.controller.ReloadComponent; +import org.apache.nifi.controller.repository.FlowFileEventRepository; +import org.apache.nifi.controller.repository.FlowFileRepository; +import org.apache.nifi.controller.scheduling.LifecycleStateManager; +import org.apache.nifi.controller.scheduling.RepositoryContextFactory; +import org.apache.nifi.controller.service.ControllerServiceProvider; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.nar.StandardExtensionDiscoveringManager; +import org.apache.nifi.nar.SystemBundle; +import org.apache.nifi.parameter.ParameterContextManager; +import org.apache.nifi.provenance.ProvenanceRepository; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.util.NiFiProperties; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mock; +import org.mockito.MockitoAnnotations; + +import java.util.Collections; +import javax.net.ssl.SSLContext; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestStandardFlowManager { + + private StandardFlowManager flowManager; + + @Mock + private NiFiProperties nifiProperties; + + @Mock + private SSLContext sslContext; + + @Mock + private FlowController flowController; + + @Mock + private FlowFileEventRepository flowFileEventRepository; + + @Mock + private ParameterContextManager parameterContextManager; + + @Mock + private ExtensionManager extensionManager; + + @Mock + private BundleCoordinate bundleCoordinate; + + @Mock + private Bundle bundle; + + @BeforeEach + public void setUp() { + MockitoAnnotations.openMocks(this); + + when(flowController.isInitialized()).thenReturn(true); + when(flowController.getExtensionManager()).thenReturn(extensionManager); + when(extensionManager.getBundle(bundleCoordinate)).thenReturn(bundle); + when(bundle.getClassLoader()).thenReturn(NopConnector.class.getClassLoader()); + + flowManager = new StandardFlowManager(nifiProperties, sslContext, flowController, flowFileEventRepository, parameterContextManager); + when(flowController.getFlowManager()).thenReturn(flowManager); + } + + + @Test + public void testCreateConnectorParameterValidation() { + final String validConnectorType = NopConnector.class.getName(); + final String validConnectorId = "test-connector-123"; + + final NullPointerException typeException = assertThrows(NullPointerException.class, () -> flowManager.createConnector(null, validConnectorId, bundleCoordinate, true, true)); + assertEquals("Connector Type", typeException.getMessage()); + + final NullPointerException idException = assertThrows(NullPointerException.class, () -> flowManager.createConnector(validConnectorType, null, bundleCoordinate, true, true)); + assertEquals("Connector ID", idException.getMessage()); + + final NullPointerException bundleException = assertThrows(NullPointerException.class, () -> flowManager.createConnector(validConnectorType, validConnectorId, null, true, true)); + assertEquals("Bundle Coordinate", bundleException.getMessage()); + } + + @Test + public void testCreateConnectorInitializesConnector() { + // Prepare a real discovering manager to provide Connector type + final StandardExtensionDiscoveringManager discoveringManager = new StandardExtensionDiscoveringManager(); + + // Discover from system bundle to pick up test services on classpath + final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties("src/test/resources/nifi-with-remote.properties"); + final Bundle systemBundle = SystemBundle.create(properties); + discoveringManager.discoverExtensions(systemBundle, Collections.emptySet()); + + // mock FlowController methods necessary for connector creation + // Most of the methods are used in order to create the ProcessGroup that will be managed by the Connector + when(flowController.getExtensionManager()).thenReturn(discoveringManager); + when(flowController.getStateManagerProvider()).thenReturn(new MockStateManagerProvider()); + final RepositoryContextFactory repositoryContextFactory = mock(RepositoryContextFactory.class); + when(repositoryContextFactory.getFlowFileRepository()).thenReturn(mock(FlowFileRepository.class)); + when(flowController.getRepositoryContextFactory()).thenReturn(repositoryContextFactory); + when(flowController.getGarbageCollectionLog()).thenReturn(mock(GarbageCollectionLog.class)); + when(flowController.getControllerServiceProvider()).thenReturn(mock(ControllerServiceProvider.class)); + when(flowController.getProvenanceRepository()).thenReturn(mock(ProvenanceRepository.class)); + when(flowController.getBulletinRepository()).thenReturn(mock(BulletinRepository.class)); + when(flowController.getLifecycleStateManager()).thenReturn(mock(LifecycleStateManager.class)); + when(flowController.getFlowFileEventRepository()).thenReturn(mock(FlowFileEventRepository.class)); + when(flowController.getReloadComponent()).thenReturn(mock(ReloadComponent.class)); + + final ConnectorValidationTrigger validationTrigger = mock(ConnectorValidationTrigger.class); + when(flowController.getConnectorValidationTrigger()).thenReturn(validationTrigger); + + final ConnectorRepository connectorRepository = mock(ConnectorRepository.class); + when(connectorRepository.createInitializationContextBuilder()).thenAnswer( + invocation -> new StandardConnectorInitializationContext.Builder()); + when(flowController.getConnectorRepository()).thenReturn(connectorRepository); + when(connectorRepository.createStateTransition(anyString(), anyString())).thenReturn(new StandardConnectorStateTransition("test component")); + + // Create the connector + final String type = NopConnector.class.getName(); + final String id = "connector-init-test"; + final ConnectorNode connectorNode = flowManager.createConnector(type, id, systemBundle.getBundleDetails().getCoordinate(), true, false); + + // Verify initialized via context presence + final Connector connector = connectorNode.getConnector(); + assertInstanceOf(NopConnector.class, connector); + + final NopConnector nopConnector = (NopConnector) connector; + assertTrue(nopConnector.isInitialized()); + assertFalse(nopConnector.isStarted()); + assertFalse(nopConnector.isConfigured()); + + final ConnectorInitializationContext initializationContext = nopConnector.getContext(); + assertNotNull(initializationContext.getLogger()); + assertEquals(id, initializationContext.getIdentifier()); + assertEquals(NopConnector.class.getSimpleName(), initializationContext.getName()); + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSocketLoadBalancedFlowFileQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSocketLoadBalancedFlowFileQueue.java index 3736856ef710..5d108bcf3dba 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSocketLoadBalancedFlowFileQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSocketLoadBalancedFlowFileQueue.java @@ -21,6 +21,7 @@ import org.apache.nifi.cluster.coordination.ClusterTopologyEventListener; import org.apache.nifi.cluster.coordination.node.NodeConnectionState; import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.connectable.Connection; import org.apache.nifi.controller.MockFlowFileRecord; import org.apache.nifi.controller.MockSwapManager; @@ -33,11 +34,16 @@ import org.apache.nifi.controller.repository.ContentRepository; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.controller.repository.FlowFileRepository; +import org.apache.nifi.controller.repository.RepositoryRecord; +import org.apache.nifi.controller.repository.RepositoryRecordType; import org.apache.nifi.controller.repository.SwapSummary; import org.apache.nifi.controller.status.FlowFileAvailability; import org.apache.nifi.events.EventReporter; import org.apache.nifi.flowfile.FlowFilePrioritizer; +import org.apache.nifi.provenance.ProvenanceEventRecord; import org.apache.nifi.provenance.ProvenanceEventRepository; +import org.apache.nifi.provenance.ProvenanceEventType; +import org.apache.nifi.provenance.StandardProvenanceEventRecord; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Timeout; @@ -47,6 +53,7 @@ import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; @@ -80,9 +87,16 @@ public class TestSocketLoadBalancedFlowFileQueue { private List nodeIds; private int nodePort = 4096; + private List repoRecords = new ArrayList<>(); + private List provRecords = new ArrayList<>(); + @BeforeEach - public void setup() { + @SuppressWarnings("unchecked") + public void setup() throws Exception { MockFlowFileRecord.resetIdGenerator(); + repoRecords.clear(); + provRecords.clear(); + Connection connection = mock(Connection.class); when(connection.getIdentifier()).thenReturn("unit-test"); @@ -109,6 +123,21 @@ public void setup() { return null; }).when(clusterCoordinator).registerEventListener(Mockito.any(ClusterTopologyEventListener.class)); + when(provRepo.eventBuilder()).thenReturn(new StandardProvenanceEventRecord.Builder()); + doAnswer((Answer) invocation -> { + final Iterable iterable = (Iterable) invocation.getArguments()[0]; + for (final ProvenanceEventRecord record : iterable) { + provRecords.add(record); + } + return null; + }).when(provRepo).registerEvents(Mockito.any(Iterable.class)); + + doAnswer((Answer) invocation -> { + final Collection records = (Collection) invocation.getArguments()[0]; + repoRecords.addAll(records); + return null; + }).when(flowFileRepo).updateRepository(Mockito.any(Collection.class)); + final ProcessScheduler scheduler = mock(ProcessScheduler.class); final AsyncLoadBalanceClientRegistry registry = mock(AsyncLoadBalanceClientRegistry.class); @@ -649,4 +678,90 @@ public boolean isRebalanceOnFailure() { return false; } } + + @Test + public void testSelectiveDropCreatesDeleteRecordsAndProvenanceEvents() throws Exception { + for (int i = 0; i < 10; i++) { + queue.put(new MockFlowFileRecord(i)); + } + + final DropFlowFileSummary summary = queue.dropFlowFiles(ff -> ff.getSize() < 5); + assertEquals(5, summary.getDroppedCount()); + + final long deleteRecordCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.DELETE).count(); + assertEquals(5, deleteRecordCount); + + assertEquals(5, provRecords.size()); + for (final ProvenanceEventRecord event : provRecords) { + assertEquals(ProvenanceEventType.DROP, event.getEventType()); + } + + assertEquals(5, queue.size().getObjectCount()); + } + + @Test + public void testSelectiveDropWithSwappedFlowFilesCreatesSwapFileRenamedRecords() throws Exception { + for (int i = 0; i < 20000; i++) { + queue.put(new MockFlowFileRecord(i % 10)); + } + + assertEquals(1, swapManager.swappedOut.size()); + + repoRecords.clear(); + provRecords.clear(); + + final DropFlowFileSummary summary = queue.dropFlowFiles(ff -> ff.getSize() < 5); + assertEquals(10000, summary.getDroppedCount()); + + final long deleteRecordCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.DELETE).count(); + assertEquals(10000, deleteRecordCount); + + final long swapFileRenamedCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_RENAMED).count(); + assertEquals(1, swapFileRenamedCount); + + final RepositoryRecord swapRenamedRecord = repoRecords.stream() + .filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_RENAMED) + .findFirst() + .orElseThrow(); + assertNotNull(swapRenamedRecord.getOriginalSwapLocation()); + assertNotNull(swapRenamedRecord.getSwapLocation()); + + assertEquals(10000, provRecords.size()); + for (final ProvenanceEventRecord event : provRecords) { + assertEquals(ProvenanceEventType.DROP, event.getEventType()); + } + + assertEquals(10000, queue.size().getObjectCount()); + } + + @Test + public void testSelectiveDropWithAllSwappedFlowFilesCreatesSwapFileDeletedRecords() throws Exception { + for (int i = 0; i < 20000; i++) { + queue.put(new MockFlowFileRecord(1)); + } + + assertEquals(1, swapManager.swappedOut.size()); + + repoRecords.clear(); + provRecords.clear(); + + final DropFlowFileSummary summary = queue.dropFlowFiles(ff -> ff.getSize() == 1); + assertEquals(20000, summary.getDroppedCount()); + + final long deleteRecordCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.DELETE).count(); + assertEquals(20000, deleteRecordCount); + + final long swapFileDeletedCount = repoRecords.stream().filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_DELETED).count(); + assertEquals(1, swapFileDeletedCount); + + final RepositoryRecord swapDeletedRecord = repoRecords.stream() + .filter(r -> r.getType() == RepositoryRecordType.SWAP_FILE_DELETED) + .findFirst() + .orElseThrow(); + assertNotNull(swapDeletedRecord.getSwapLocation()); + + assertEquals(20000, provRecords.size()); + + assertEquals(0, queue.size().getObjectCount()); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSwappablePriorityQueue.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSwappablePriorityQueue.java index 382f2ae35668..05550aaa3b8c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSwappablePriorityQueue.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/queue/clustered/TestSwappablePriorityQueue.java @@ -24,6 +24,7 @@ import org.apache.nifi.controller.queue.FlowFileQueue; import org.apache.nifi.controller.queue.PollStrategy; import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.controller.queue.SelectiveDropResult; import org.apache.nifi.controller.queue.SwappablePriorityQueue; import org.apache.nifi.controller.repository.FlowFileRecord; import org.apache.nifi.events.EventReporter; @@ -45,6 +46,7 @@ import java.util.List; import java.util.Map; import java.util.Set; +import java.util.function.Predicate; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; @@ -547,24 +549,25 @@ public void testSwapIn() { @Test public void testSwapInWhenThresholdIsLessThanSwapSize() { // create a queue where the swap threshold is less than 10k + // With threshold of 1000, swapRecordPollSize will also be 1000 (min of 10000 and threshold) queue = new SwappablePriorityQueue(swapManager, 1000, eventReporter, flowFileQueue, dropAction, null); for (int i = 1; i <= 20000; i++) { queue.put(new MockFlowFileRecord()); } - assertEquals(1, swapManager.swappedOut.size()); + // With swapRecordPollSize = 1000, we expect 19 swap files (19,000 FlowFiles) + assertEquals(19, swapManager.swappedOut.size()); queue.put(new MockFlowFileRecord()); - assertEquals(1, swapManager.swappedOut.size()); + assertEquals(19, swapManager.swappedOut.size()); final Set exp = new HashSet<>(); // At this point there should be: - // 1k FlowFiles in the active queue - // 9,001 FlowFiles in the swap queue - // 10k FlowFiles swapped to disk - - for (int i = 0; i < 999; i++) { // + // 1k flow files in the active queue + // 1 flow file in the swap queue + // 19k flow files swapped to disk (19 swap files with 1k each) + for (int i = 0; i < 999; i++) { final FlowFileRecord flowFile = queue.poll(exp, 500000); assertNotNull(flowFile); assertEquals(1, queue.getQueueDiagnostics().getUnacknowledgedQueueSize().getObjectCount()); @@ -582,13 +585,13 @@ public void testSwapInWhenThresholdIsLessThanSwapSize() { assertEquals(0, swapManager.swapInCalledCount); assertEquals(0, queue.getQueueDiagnostics().getActiveQueueSize().getObjectCount()); - assertEquals(1, swapManager.swapOutCalledCount); + assertEquals(19, swapManager.swapOutCalledCount); - assertNotNull(queue.poll(exp, 500000)); // this should trigger a swap-in of 10,000 records, and then pull 1 off the top. + assertNotNull(queue.poll(exp, 500000)); // this should trigger a swap-in of 1,000 records, and then pull 1 off the top. assertEquals(1, swapManager.swapInCalledCount); - assertEquals(9999, queue.getQueueDiagnostics().getActiveQueueSize().getObjectCount()); + assertEquals(999, queue.getQueueDiagnostics().getActiveQueueSize().getObjectCount()); - assertTrue(swapManager.swappedOut.isEmpty()); + assertEquals(18, swapManager.swappedOut.size()); queue.poll(exp, 500000); } @@ -672,6 +675,101 @@ public void testDropSwappedFlowFiles() { assertEquals(2, swapManager.swapInCalledCount); } + @Test + @Timeout(120) + public void testSelectiveDropFromActiveQueue() throws IOException { + final Predicate evenSizePredicate = flowFile -> flowFile.getSize() % 2 == 0; + + for (int i = 0; i < 100; i++) { + queue.put(new MockFlowFileRecord(i)); + } + assertEquals(100, queue.size().getObjectCount()); + + final SelectiveDropResult result = queue.dropFlowFiles(evenSizePredicate); + + assertEquals(50, result.getDroppedCount()); + assertEquals(50, queue.size().getObjectCount()); + + final Set expired = new HashSet<>(); + for (int i = 0; i < 50; i++) { + final FlowFileRecord flowFile = queue.poll(expired, 0L); + assertNotNull(flowFile); + assertEquals(1, flowFile.getSize() % 2); + } + assertNull(queue.poll(expired, 0L)); + } + + @Test + @Timeout(120) + public void testSelectiveDropFromSwapQueue() throws IOException { + final Predicate evenSizePredicate = flowFile -> flowFile.getSize() % 2 == 0; + + for (int i = 0; i < 15000; i++) { + queue.put(new MockFlowFileRecord(i)); + } + assertEquals(15000, queue.size().getObjectCount()); + assertEquals(10000, queue.getQueueDiagnostics().getActiveQueueSize().getObjectCount()); + + final SelectiveDropResult result = queue.dropFlowFiles(evenSizePredicate); + + assertEquals(7500, result.getDroppedCount()); + assertEquals(7500, queue.size().getObjectCount()); + } + + @Test + @Timeout(120) + public void testSelectiveDropFromSwapFile() throws IOException { + final Predicate evenSizePredicate = flowFile -> flowFile.getSize() % 2 == 0; + + for (int i = 0; i < 30000; i++) { + queue.put(new MockFlowFileRecord(i)); + } + assertEquals(30000, queue.size().getObjectCount()); + assertEquals(2, swapManager.swappedOut.size()); + + final SelectiveDropResult result = queue.dropFlowFiles(evenSizePredicate); + + assertEquals(15000, result.getDroppedCount()); + assertEquals(15000, queue.size().getObjectCount()); + assertEquals(2, result.getSwapLocationUpdates().size()); + } + + @Test + @Timeout(120) + public void testSelectiveDropRemovesEntireSwapFileWhenAllMatch() throws IOException { + final Predicate allMatch = flowFile -> true; + + for (int i = 0; i < 30000; i++) { + queue.put(new MockFlowFileRecord(i)); + } + assertEquals(30000, queue.size().getObjectCount()); + assertEquals(2, swapManager.swappedOut.size()); + + final SelectiveDropResult result = queue.dropFlowFiles(allMatch); + + assertEquals(30000, result.getDroppedCount()); + assertEquals(0, queue.size().getObjectCount()); + + long nullNewLocations = result.getSwapLocationUpdates().values().stream().filter(v -> v == null).count(); + assertEquals(2, nullNewLocations); + } + + @Test + @Timeout(120) + public void testSelectiveDropWithNoMatches() throws IOException { + final Predicate noMatch = flowFile -> false; + + for (int i = 0; i < 100; i++) { + queue.put(new MockFlowFileRecord(i)); + } + assertEquals(100, queue.size().getObjectCount()); + + final SelectiveDropResult result = queue.dropFlowFiles(noMatch); + + assertEquals(0, result.getDroppedCount()); + assertEquals(100, queue.size().getObjectCount()); + } + @Test @Timeout(5) public void testGetActiveFlowFilesReturnsAllActiveFlowFiles() throws InterruptedException { @@ -746,6 +844,7 @@ public void testOOMEFollowedBySuccessfulSwapIn() { // To truly test this we need to get both the in-memory swap queue and swap "on disk" involved. public void testLastQueueDateMetrics() throws IOException { Set flowFileRecords = new HashSet<>(11001); + // With threshold of 1000, swapRecordPollSize will also be 1000 queue = new SwappablePriorityQueue(swapManager, 1000, eventReporter, flowFileQueue, dropAction, "testGetMinLastQueueDate"); long minQueueDate = Long.MAX_VALUE; long totalQueueDate = 0L; @@ -759,10 +858,11 @@ public void testLastQueueDateMetrics() throws IOException { } // Assert the queue has a max of active, in-memory swap, and on-disk swap + // With swapRecordPollSize = 1000, we have 10 swap files (10,000 FlowFiles) + 1 in swap queue + 1,000 in active assertEquals(1000, queue.getActiveFlowFiles().size()); assertEquals(10001, queue.getFlowFileQueueSize().getSwappedCount()); - assertEquals(1, queue.getFlowFileQueueSize().getSwapFileCount()); - assertEquals(10000, swapManager.getSwapSummary(swapManager.recoverSwapLocations(flowFileQueue, "testGetMinLastQueueDate").get(0)).getQueueSize().getObjectCount()); + assertEquals(10, queue.getFlowFileQueueSize().getSwapFileCount()); + assertEquals(1000, swapManager.getSwapSummary(swapManager.recoverSwapLocations(flowFileQueue, "testGetMinLastQueueDate").get(0)).getQueueSize().getObjectCount()); // Ensure that the min and totals are correct long now = System.currentTimeMillis(); @@ -773,11 +873,13 @@ public void testLastQueueDateMetrics() throws IOException { List polledRecords = queue.poll(1000, Collections.emptySet(), -1); polledRecords.addAll(queue.poll(2, Collections.emptySet(), -1)); - // Assert that the lone swap file was recovered into memory and that all numbers are as we still expect them to be. - assertEquals(9998, queue.getActiveFlowFiles().size()); - assertEquals(1, queue.getFlowFileQueueSize().getSwappedCount()); - assertEquals(0, queue.getFlowFileQueueSize().getSwapFileCount()); - assertTrue(swapManager.recoverSwapLocations(flowFileQueue, "testGetMinLastQueueDate").isEmpty()); + // Assert that one swap file was recovered into memory and that all numbers are as we still expect them to be. + // After polling 1,002: 1,000 active polled, then swap-in of 1,000 (from first swap file), then 2 more polled + // So: 998 active, 1 in swap queue, 9 swap files remaining + assertEquals(998, queue.getActiveFlowFiles().size()); + assertEquals(9001, queue.getFlowFileQueueSize().getSwappedCount()); + assertEquals(9, queue.getFlowFileQueueSize().getSwapFileCount()); + assertEquals(9, swapManager.recoverSwapLocations(flowFileQueue, "testGetMinLastQueueDate").size()); // Ensure that the min and total are still correct flowFileRecords.removeAll(polledRecords); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerdeTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerdeTest.java index e11c20be6ac6..936979f79dfa 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerdeTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/SchemaRepositoryRecordSerdeTest.java @@ -155,7 +155,7 @@ public void testV1SwapInCantHandleLongAttributeName() throws IOException { } attributes.put(stringBuilder.toString(), "testValue"); StandardRepositoryRecord record = createCreateFlowFileRecord(attributes); - record.setSwapLocation("fake"); + record.setSwapLocation("fake", SWAP_IN); assertEquals(SWAP_IN, record.getType()); schemaRepositoryRecordSerde.serializeRecord(new LiveSerializedRepositoryRecord(record), dataOutputStream, RepositoryRecordSchema.SWAP_IN_SCHEMA_V1, RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V1); @@ -176,7 +176,7 @@ public void testV1SwapInCantHandleLongAttributeValue() throws IOException { } attributes.put("testName", stringBuilder.toString()); StandardRepositoryRecord record = createCreateFlowFileRecord(attributes); - record.setSwapLocation("fake"); + record.setSwapLocation("fake", SWAP_IN); assertEquals(SWAP_IN, record.getType()); schemaRepositoryRecordSerde.serializeRecord(new LiveSerializedRepositoryRecord(record), dataOutputStream, RepositoryRecordSchema.SWAP_IN_SCHEMA_V1, RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V1); @@ -197,7 +197,7 @@ public void testV2SwapInCanHandleLongAttributeName() throws IOException { } attributes.put(stringBuilder.toString(), "testValue"); StandardRepositoryRecord record = createCreateFlowFileRecord(attributes); - record.setSwapLocation("fake"); + record.setSwapLocation("fake", SWAP_IN); assertEquals(SWAP_IN, record.getType()); schemaRepositoryRecordSerde.serializeRecord(new LiveSerializedRepositoryRecord(record), dataOutputStream); @@ -217,7 +217,7 @@ public void testV2SwapInCanHandleLongAttributeValue() throws IOException { } attributes.put("testName", stringBuilder.toString()); StandardRepositoryRecord record = createCreateFlowFileRecord(attributes); - record.setSwapLocation("fake"); + record.setSwapLocation("fake", SWAP_IN); assertEquals(SWAP_IN, record.getType()); schemaRepositoryRecordSerde.serializeRecord(new LiveSerializedRepositoryRecord(record), dataOutputStream); @@ -233,7 +233,7 @@ public void testRoundTripSwapInV1ToV2() throws IOException { Map attributes = new HashMap<>(); attributes.put("testName", "testValue"); StandardRepositoryRecord record = createCreateFlowFileRecord(attributes); - record.setSwapLocation("fake"); + record.setSwapLocation("fake", SWAP_IN); assertEquals(SWAP_IN, record.getType()); schemaRepositoryRecordSerde.serializeRecord(new LiveSerializedRepositoryRecord(record), dataOutputStream, RepositoryRecordSchema.SWAP_IN_SCHEMA_V1, RepositoryRecordSchema.REPOSITORY_RECORD_SCHEMA_V1); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java index 8eb5444e47ea..7835da135301 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java @@ -20,6 +20,7 @@ import org.apache.nifi.components.state.Scope; import org.apache.nifi.components.state.StateMap; import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.ConnectableFlowFileActivity; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; import org.apache.nifi.controller.BackoffMechanism; @@ -199,6 +200,7 @@ public void setup() throws IOException { when(connectable.getComponentType()).thenReturn("Unit Test Component"); when(connectable.getBackoffMechanism()).thenReturn(BackoffMechanism.PENALIZE_FLOWFILE); when(connectable.getMaxBackoffPeriod()).thenReturn("1 sec"); + when(connectable.getFlowFileActivity()).thenReturn(new ConnectableFlowFileActivity()); Mockito.doAnswer((Answer>) invocation -> { final Object[] arguments = invocation.getArguments(); @@ -213,7 +215,7 @@ public void setup() throws IOException { }).when(connectable).getConnections(Mockito.any(Relationship.class)); when(connectable.getConnections()).thenReturn(new HashSet<>(connList)); - + when(connectable.getFlowFileActivity()).thenReturn(new ConnectableFlowFileActivity()); contentRepo = new MockContentRepository(); contentRepo.initialize(new StandardContentRepositoryContext(new StandardResourceClaimManager(), EventReporter.NO_OP)); flowFileRepo = new MockFlowFileRepository(contentRepo); @@ -2958,6 +2960,7 @@ public Connectable createProcessorConnectable() { when(connectable.getIdentifier()).thenReturn("connectable-1"); when(connectable.getConnectableType()).thenReturn(ConnectableType.PROCESSOR); when(connectable.getComponentType()).thenReturn("Unit Test Component"); + when(connectable.getFlowFileActivity()).thenReturn(new ConnectableFlowFileActivity()); Mockito.doAnswer((Answer>) invocation -> { final Object[] arguments = invocation.getArguments(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestWriteAheadFlowFileRepository.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestWriteAheadFlowFileRepository.java index 9d6118e8ebe6..dcbbd1e80c40 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestWriteAheadFlowFileRepository.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestWriteAheadFlowFileRepository.java @@ -16,6 +16,7 @@ */ package org.apache.nifi.controller.repository; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Connection; import org.apache.nifi.controller.MockFlowFileRecord; @@ -39,6 +40,7 @@ import org.apache.nifi.controller.status.FlowFileAvailability; import org.apache.nifi.controller.swap.StandardSwapContents; import org.apache.nifi.controller.swap.StandardSwapSummary; +import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.processor.FlowFileFilter; import org.apache.nifi.util.MockFlowFile; @@ -69,6 +71,7 @@ import java.util.Set; import java.util.UUID; import java.util.concurrent.TimeUnit; +import java.util.function.Predicate; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; @@ -287,6 +290,11 @@ public DropFlowFileStatus cancelDropFlowFileRequest(String requestIdentifier) { return null; } + @Override + public DropFlowFileSummary dropFlowFiles(Predicate predicate) { + return new DropFlowFileSummary(0, 0L); + } + @Override public ListFlowFileStatus listFlowFiles(String requestIdentifier, int maxResults) { return null; @@ -314,7 +322,7 @@ public void verifyCanList() throws IllegalStateException { @Override public QueueDiagnostics getQueueDiagnostics() { final FlowFileQueueSize size = new FlowFileQueueSize(size().getObjectCount(), size().getByteCount(), 0, 0, 0, 0, 0); - return new StandardQueueDiagnostics(new StandardLocalQueuePartitionDiagnostics(size, false, false), Collections.emptyList()); + return new StandardQueueDiagnostics(new StandardLocalQueuePartitionDiagnostics(size, false, false, new QueueSize(0, 0), new QueueSize(0, 0)), Collections.emptyList()); } @Override @@ -682,6 +690,126 @@ public void testRestartWithOneRecord() throws IOException { repo2.close(); } + @Test + public void testSelectiveDropWithSwappedFlowFilesAndRecovery() throws IOException { + final Path path = Paths.get("target/test-repo"); + if (Files.exists(path)) { + FileUtils.deleteFile(path.toFile(), true); + } + + final ResourceClaimManager claimManager = new StandardResourceClaimManager(); + final TestQueueProvider queueProvider = new TestQueueProvider(); + final Connection connection = Mockito.mock(Connection.class); + when(connection.getIdentifier()).thenReturn("selective-drop-test"); + when(connection.getDestination()).thenReturn(Mockito.mock(Connectable.class)); + + final MockFlowFileSwapManager swapMgr = new MockFlowFileSwapManager(); + final FlowFileQueue queue = new StandardFlowFileQueue("selective-drop-test", null, null, null, swapMgr, null, 10000, "0 sec", 0L, "0 B"); + when(connection.getFlowFileQueue()).thenReturn(queue); + queueProvider.addConnection(connection); + + final List allFlowFiles = new ArrayList<>(); + final String originalSwapLocation; + final String newSwapLocation; + + try (final WriteAheadFlowFileRepository repo = new WriteAheadFlowFileRepository(niFiProperties)) { + repo.initialize(claimManager); + repo.loadFlowFiles(queueProvider); + + for (int i = 0; i < 20; i++) { + final FlowFileRecord flowFile = new StandardFlowFileRecord.Builder() + .id(i + 1) + .addAttribute("uuid", UUID.randomUUID().toString()) + .addAttribute("index", String.valueOf(i)) + .size(i) + .build(); + + final StandardRepositoryRecord record = new StandardRepositoryRecord(queue); + record.setWorking(flowFile, false); + record.setDestination(queue); + repo.updateRepository(Collections.singletonList(record)); + allFlowFiles.add(flowFile); + } + + final List flowFilesToSwap = new ArrayList<>(); + for (int i = 10; i < 20; i++) { + flowFilesToSwap.add(allFlowFiles.get(i)); + } + originalSwapLocation = swapMgr.swapOut(flowFilesToSwap, queue, null); + repo.swapFlowFilesOut(flowFilesToSwap, queue, originalSwapLocation); + + final List selectiveDropRecords = new ArrayList<>(); + + for (int i = 0; i < 5; i++) { + final FlowFileRecord ff = allFlowFiles.get(i); + final StandardRepositoryRecord dropRecord = new StandardRepositoryRecord(queue, ff); + dropRecord.markForDelete(); + selectiveDropRecords.add(dropRecord); + } + + for (int i = 10; i < 15; i++) { + final FlowFileRecord ff = allFlowFiles.get(i); + final StandardRepositoryRecord dropRecord = new StandardRepositoryRecord(queue, ff); + dropRecord.markForDelete(); + selectiveDropRecords.add(dropRecord); + } + + final List remainingSwapped = new ArrayList<>(); + for (int i = 15; i < 20; i++) { + remainingSwapped.add(allFlowFiles.get(i)); + } + newSwapLocation = swapMgr.swapOut(remainingSwapped, queue, null); + + for (final FlowFileRecord ff : remainingSwapped) { + final StandardRepositoryRecord swapRecord = new StandardRepositoryRecord(queue, ff, newSwapLocation); + swapRecord.setDestination(queue); + selectiveDropRecords.add(swapRecord); + } + + final StandardRepositoryRecord swapFileRenamedRecord = new StandardRepositoryRecord(queue); + swapFileRenamedRecord.setSwapFileRenamed(originalSwapLocation, newSwapLocation); + selectiveDropRecords.add(swapFileRenamedRecord); + + repo.updateRepository(selectiveDropRecords); + + swapMgr.deleteSwapFile(originalSwapLocation); + } + + final List recoveredFlowFiles = new ArrayList<>(); + final ResourceClaimManager recoveryClaimManager = new StandardResourceClaimManager(); + + final Connection recoveryConnection = Mockito.mock(Connection.class); + when(recoveryConnection.getIdentifier()).thenReturn("selective-drop-test"); + when(recoveryConnection.getDestination()).thenReturn(Mockito.mock(Connectable.class)); + + final FlowFileQueue recoveryQueue = Mockito.mock(FlowFileQueue.class); + when(recoveryQueue.getIdentifier()).thenReturn("selective-drop-test"); + doAnswer(invocation -> { + recoveredFlowFiles.add((FlowFileRecord) invocation.getArguments()[0]); + return null; + }).when(recoveryQueue).put(any(FlowFileRecord.class)); + + when(recoveryConnection.getFlowFileQueue()).thenReturn(recoveryQueue); + + final TestQueueProvider recoveryQueueProvider = new TestQueueProvider(); + recoveryQueueProvider.addConnection(recoveryConnection); + + try (final WriteAheadFlowFileRepository repo = new WriteAheadFlowFileRepository(niFiProperties)) { + repo.initialize(recoveryClaimManager); + repo.loadFlowFiles(recoveryQueueProvider); + + assertEquals(5, recoveredFlowFiles.size()); + + for (final FlowFileRecord ff : recoveredFlowFiles) { + final int index = Integer.parseInt(ff.getAttribute("index")); + assertTrue(index >= 5 && index < 10, "Expected active FlowFiles with index 5-9, but found index " + index); + } + + assertFalse(repo.isValidSwapLocationSuffix(originalSwapLocation), "Original swap location should not be valid after selective drop"); + assertTrue(repo.isValidSwapLocationSuffix(newSwapLocation), "New swap location should be valid after selective drop"); + } + } + private static class TestQueueProvider implements QueueProvider { private List connectionList = new ArrayList<>(); @@ -795,6 +923,13 @@ public void purge() { this.swappedRecords.clear(); } + @Override + public void deleteSwapFile(final String swapLocation) { + for (final Map> swapMap : swappedRecords.values()) { + swapMap.remove(swapLocation); + } + } + @Override public String getQueueIdentifier(final String swapLocation) { return null; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestStandardProcessScheduler.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestStandardProcessScheduler.java index dd2c6ae70e9e..abadd3b0477b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestStandardProcessScheduler.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestStandardProcessScheduler.java @@ -48,6 +48,7 @@ import org.apache.nifi.controller.service.ControllerServiceNode; import org.apache.nifi.controller.service.ControllerServiceProvider; import org.apache.nifi.controller.service.ControllerServiceState; +import org.apache.nifi.controller.service.StandardConfigurationContext; import org.apache.nifi.controller.service.StandardControllerServiceProvider; import org.apache.nifi.controller.service.mock.MockProcessGroup; import org.apache.nifi.engine.FlowEngine; @@ -64,6 +65,7 @@ import org.apache.nifi.processor.StandardProcessorInitializationContext; import org.apache.nifi.processor.StandardValidationContextFactory; import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.reporting.AbstractReportingTask; import org.apache.nifi.reporting.InitializationException; import org.apache.nifi.reporting.ReportingContext; @@ -88,6 +90,7 @@ import java.util.Map; import java.util.Random; import java.util.UUID; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.ExecutionException; @@ -95,6 +98,7 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; @@ -491,6 +495,51 @@ public void validateNeverEnablingServiceCanStillBeDisabled() throws Exception { assertEquals(0, ts.disableInvocationCount()); } + @Test + @Timeout(10) + public void testEnableControllerServiceWithConfigurationContext() throws Exception { + final ControllerServiceNode serviceNode = flowManager.createControllerService(PropertyTrackingService.class.getName(), + "property-tracking-service", systemBundle.getBundleDetails().getCoordinate(), null, false, true, null); + + rootGroup.addControllerService(serviceNode); + serviceNode.setProperties(Map.of(PropertyTrackingService.TRACKING_PROPERTY.getName(), "original-value")); + serviceNode.performValidation(); + + final ConfigurationContext overrideContext = new StandardConfigurationContext( + serviceNode, Map.of(PropertyTrackingService.TRACKING_PROPERTY.getName(), "overridden-value"), null, + rootGroup.getParameterContext(), serviceProvider, null); + + final CompletableFuture future = scheduler.enableControllerService(serviceNode, overrideContext); + future.get(5, TimeUnit.SECONDS); + + final PropertyTrackingService service = (PropertyTrackingService) serviceNode.getControllerServiceImplementation(); + assertEquals(1, service.enableInvocationCount()); + assertEquals("overridden-value", service.getEnabledPropertyValue()); + assertEquals(ControllerServiceState.ENABLED, serviceNode.getState()); + } + + @Test + @Timeout(10) + public void testEnableControllerServiceWithConfigurationContextUsesOverriddenProperties() throws ExecutionException, InterruptedException, TimeoutException { + final ControllerServiceNode serviceNode = flowManager.createControllerService(PropertyTrackingService.class.getName(), + "property-tracking-service-2", systemBundle.getBundleDetails().getCoordinate(), null, false, true, null); + + rootGroup.addControllerService(serviceNode); + serviceNode.performValidation(); + + final ConfigurationContext validOverrideContext = new StandardConfigurationContext( + serviceNode, Map.of(PropertyTrackingService.TRACKING_PROPERTY.getName(), "override-value"), null, + rootGroup.getParameterContext(), serviceProvider, null); + + final CompletableFuture future = scheduler.enableControllerService(serviceNode, validOverrideContext); + future.get(5, TimeUnit.SECONDS); + + final PropertyTrackingService service = (PropertyTrackingService) serviceNode.getControllerServiceImplementation(); + assertEquals(1, service.enableInvocationCount()); + assertEquals("override-value", service.getEnabledPropertyValue()); + assertEquals(ControllerServiceState.ENABLED, serviceNode.getState()); + } + // Test that if processor throws Exception in @OnScheduled, it keeps getting scheduled @Test @Timeout(10) @@ -626,6 +675,38 @@ public int disableInvocationCount() { } } + public static class PropertyTrackingService extends AbstractControllerService { + public static final PropertyDescriptor TRACKING_PROPERTY = new PropertyDescriptor.Builder() + .name("Tracking Property") + .description("A property for tracking what value was used during enabling") + .required(false) + .defaultValue("default-value") + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + private volatile String enabledPropertyValue; + private final AtomicInteger enableCounter = new AtomicInteger(); + + @Override + protected List getSupportedPropertyDescriptors() { + return List.of(TRACKING_PROPERTY); + } + + @OnEnabled + public void enable(final ConfigurationContext context) { + this.enabledPropertyValue = context.getProperty(TRACKING_PROPERTY).getValue(); + this.enableCounter.incrementAndGet(); + } + + public String getEnabledPropertyValue() { + return enabledPropertyValue; + } + + public int enableInvocationCount() { + return enableCounter.get(); + } + } + private StandardProcessScheduler createScheduler() { return new StandardProcessScheduler(new FlowEngine(1, "Unit Test", true), Mockito.mock(FlowController.class), stateMgrProvider, nifiProperties, new StandardLifecycleStateManager()); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizerTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizerTest.java index b01317c83ecc..f526e25cb444 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizerTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizerTest.java @@ -18,6 +18,7 @@ import org.apache.nifi.cluster.protocol.DataFlow; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorRepository; import org.apache.nifi.controller.FlowController; import org.apache.nifi.controller.ReportingTaskNode; import org.apache.nifi.controller.SnippetManager; @@ -51,6 +52,7 @@ import java.io.File; import java.nio.charset.StandardCharsets; import java.nio.file.Path; +import java.util.Collections; import java.util.List; import java.util.Map; import java.util.UUID; @@ -278,5 +280,9 @@ private void setFlowController() { when(flowController.getEncryptor()).thenReturn(encryptor); when(flowController.createVersionedComponentStateLookup(any())).thenReturn(stateLookup); when(flowController.getControllerServiceProvider()).thenReturn(controllerServiceProvider); + + final ConnectorRepository connectorRepository = mock(ConnectorRepository.class); + when(connectorRepository.getConnectors()).thenReturn(Collections.emptyList()); + when(flowController.getConnectorRepository()).thenReturn(connectorRepository); } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/TestStandardControllerServiceProvider.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/TestStandardControllerServiceProvider.java index 0e81a6e7bb98..217caa0a5416 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/TestStandardControllerServiceProvider.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/TestStandardControllerServiceProvider.java @@ -20,10 +20,8 @@ import org.apache.nifi.bundle.Bundle; import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.components.PropertyDescriptor; -import org.apache.nifi.components.state.Scope; import org.apache.nifi.components.state.StateManager; import org.apache.nifi.components.state.StateManagerProvider; -import org.apache.nifi.components.state.StateMap; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.components.validation.ValidationTrigger; import org.apache.nifi.components.validation.VerifiableComponentFactory; @@ -31,6 +29,7 @@ import org.apache.nifi.controller.ExtensionBuilder; import org.apache.nifi.controller.FlowController; import org.apache.nifi.controller.LoggableComponent; +import org.apache.nifi.controller.MockStateManagerProvider; import org.apache.nifi.controller.NodeTypeProvider; import org.apache.nifi.controller.ProcessScheduler; import org.apache.nifi.controller.ProcessorNode; @@ -45,7 +44,6 @@ import org.apache.nifi.controller.service.mock.ServiceA; import org.apache.nifi.controller.service.mock.ServiceB; import org.apache.nifi.controller.service.mock.ServiceC; -import org.apache.nifi.controller.state.StandardStateMap; import org.apache.nifi.engine.FlowEngine; import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.nar.ExtensionDiscoveringManager; @@ -64,14 +62,12 @@ import org.mockito.Mockito; import org.mockito.stubbing.Answer; -import java.io.IOException; import java.net.URL; import java.util.Arrays; import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; -import java.util.Optional; import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; @@ -92,41 +88,7 @@ public class TestStandardControllerServiceProvider { - private static final StateManagerProvider stateManagerProvider = new StateManagerProvider() { - @Override - public StateManager getStateManager(final String componentId, final boolean dropStateKeySupported) { - final StateManager stateManager = Mockito.mock(StateManager.class); - final StateMap emptyStateMap = new StandardStateMap(Collections.emptyMap(), Optional.empty()); - try { - Mockito.when(stateManager.getState(any(Scope.class))).thenReturn(emptyStateMap); - } catch (IOException e) { - throw new AssertionError(); - } - - return stateManager; - } - - @Override - public void shutdown() { - } - - @Override - public void enableClusterProvider() { - } - - @Override - public void disableClusterProvider() { - } - - @Override - public void onComponentRemoved(final String componentId) { - } - - @Override - public boolean isClusterProviderEnabled() { - return false; - } - }; + private static final StateManagerProvider stateManagerProvider = new MockStateManagerProvider(); private static NiFiProperties niFiProperties; private static ExtensionDiscoveringManager extensionManager; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/mock/MockProcessGroup.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/mock/MockProcessGroup.java index cfd0eeefe552..6beb29d409eb 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/mock/MockProcessGroup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/service/mock/MockProcessGroup.java @@ -21,10 +21,12 @@ import org.apache.nifi.authorization.resource.Authorizable; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Connection; +import org.apache.nifi.connectable.FlowFileActivity; import org.apache.nifi.connectable.Funnel; import org.apache.nifi.connectable.Port; import org.apache.nifi.connectable.Position; import org.apache.nifi.connectable.Positionable; +import org.apache.nifi.connectable.ProcessGroupFlowFileActivity; import org.apache.nifi.controller.ProcessorNode; import org.apache.nifi.controller.Snippet; import org.apache.nifi.controller.flow.FlowManager; @@ -125,6 +127,11 @@ public void setName(final String name) { } + @Override + public Optional getConnectorIdentifier() { + return Optional.empty(); + } + @Override public void setPosition(final Position position) { @@ -713,7 +720,7 @@ public ComponentAdditions addVersionedComponents(VersionedComponentAdditions add } @Override - public void updateFlow(VersionedExternalFlow proposedFlow, String componentIdSeed, boolean verifyNotDirty, boolean updateSettings, boolean updateDescendantVerisonedFlows) { + public void updateFlow(VersionedExternalFlow proposedFlow, String componentIdSeed, boolean verifyNotDirty, boolean updateSettings, boolean updateDescendantVersionedFlows) { } @Override @@ -903,6 +910,15 @@ public String getStatelessFlowTimeout() { return null; } + @Override + public FlowFileActivity getFlowFileActivity() { + return new ProcessGroupFlowFileActivity(this); + } + + @Override + public void setExplicitParentAuthorizable(final Authorizable parent) { + } + @Override public void setLogFileSuffix(String logFileSuffix) { @@ -912,4 +928,12 @@ public void setLogFileSuffix(String logFileSuffix) { public void terminateProcessor(ProcessorNode processor) { } + @Override + public CompletableFuture purge() { + processorMap.clear(); + serviceMap.clear(); + inputPortMap.clear(); + outputPortMap.clear(); + return CompletableFuture.completedFuture(null); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/tasks/TestStatelessFlowTask.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/tasks/TestStatelessFlowTask.java index d1c122f2e5b4..5b6335082a88 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/tasks/TestStatelessFlowTask.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/tasks/TestStatelessFlowTask.java @@ -17,8 +17,11 @@ package org.apache.nifi.controller.tasks; +import org.apache.nifi.connectable.ConnectableFlowFileActivity; import org.apache.nifi.connectable.Connection; import org.apache.nifi.connectable.ConnectionUtils.FlowFileCloneResult; +import org.apache.nifi.connectable.FlowFileActivity; +import org.apache.nifi.connectable.FlowFileTransferCounts; import org.apache.nifi.connectable.Port; import org.apache.nifi.controller.MockFlowFileRecord; import org.apache.nifi.controller.queue.FlowFileQueue; @@ -63,6 +66,7 @@ import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.OptionalLong; import java.util.Set; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -90,6 +94,8 @@ public class TestStatelessFlowTask { private List statelessProvenanceEvents; private Map flowFileEventsByComponentId; private ProvenanceEventRepository statelessProvRepo; + private FlowFileActivity groupNodeFlowFileActivity; + private StatelessDataflow statelessFlow; @BeforeEach public void setup() throws IOException { @@ -123,10 +129,13 @@ public void setup() throws IOException { return statelessProvenanceEvents.subList((int) startEventId, (int) lastEvent); }).when(statelessProvRepo).getEvents(anyLong(), anyInt()); - final StatelessDataflow statelessFlow = mock(StatelessDataflow.class); + statelessFlow = mock(StatelessDataflow.class); + when(statelessFlow.getLatestActivityTime()).thenReturn(OptionalLong.empty()); final StatelessGroupNode statelessGroupNode = mock(StatelessGroupNode.class); when(statelessGroupNode.getProcessGroup()).thenReturn(rootGroup); + groupNodeFlowFileActivity = new ConnectableFlowFileActivity(); + when(statelessGroupNode.getFlowFileActivity()).thenReturn(groupNodeFlowFileActivity); final FlowFileRepository flowFileRepo = mock(FlowFileRepository.class); @@ -530,6 +539,135 @@ public void testUpdateEventRepositoryWithInputFlowFile() { assertEquals(10, inputPortEvent.getContentSizeOut()); } + + @Test + public void testUpdateFlowFileActivityWithNoEvents() { + task.updateFlowFileActivity(statelessProvRepo); + + final FlowFileTransferCounts counts = groupNodeFlowFileActivity.getTransferCounts(); + assertEquals(0L, counts.getReceivedCount()); + assertEquals(0L, counts.getReceivedBytes()); + assertEquals(0L, counts.getSentCount()); + assertEquals(0L, counts.getSentBytes()); + assertTrue(groupNodeFlowFileActivity.getLatestActivityTime().isEmpty()); + } + + @Test + public void testUpdateFlowFileActivityCountsReceiveAndSendEvents() { + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.RECEIVE, 100L)); + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.RECEIVE, 200L)); + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.SEND, 50L)); + + task.updateFlowFileActivity(statelessProvRepo); + + final FlowFileTransferCounts counts = groupNodeFlowFileActivity.getTransferCounts(); + assertEquals(2L, counts.getReceivedCount()); + assertEquals(300L, counts.getReceivedBytes()); + assertEquals(1L, counts.getSentCount()); + assertEquals(50L, counts.getSentBytes()); + } + + @Test + public void testUpdateFlowFileActivityCountsCreateEvents() { + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.CREATE, 75L)); + + task.updateFlowFileActivity(statelessProvRepo); + + final FlowFileTransferCounts counts = groupNodeFlowFileActivity.getTransferCounts(); + assertEquals(1L, counts.getReceivedCount()); + assertEquals(75L, counts.getReceivedBytes()); + assertEquals(0L, counts.getSentCount()); + assertEquals(0L, counts.getSentBytes()); + } + + @Test + public void testUpdateFlowFileActivityCountsFetchBytesButNotCount() { + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.FETCH, 500L)); + + task.updateFlowFileActivity(statelessProvRepo); + + final FlowFileTransferCounts counts = groupNodeFlowFileActivity.getTransferCounts(); + assertEquals(0L, counts.getReceivedCount()); + assertEquals(500L, counts.getReceivedBytes()); + assertEquals(0L, counts.getSentCount()); + assertEquals(0L, counts.getSentBytes()); + } + + @Test + public void testUpdateFlowFileActivityIgnoresOtherEventTypes() { + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.ATTRIBUTES_MODIFIED, 100L)); + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.CONTENT_MODIFIED, 200L)); + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.ROUTE, 300L)); + + task.updateFlowFileActivity(statelessProvRepo); + + final FlowFileTransferCounts counts = groupNodeFlowFileActivity.getTransferCounts(); + assertEquals(0L, counts.getReceivedCount()); + assertEquals(0L, counts.getReceivedBytes()); + assertEquals(0L, counts.getSentCount()); + assertEquals(0L, counts.getSentBytes()); + } + + @Test + public void testUpdateFlowFileActivitySetsActivityTimeFromFlow() { + when(statelessFlow.getLatestActivityTime()).thenReturn(OptionalLong.of(System.currentTimeMillis())); + + task.updateFlowFileActivity(statelessProvRepo); + + assertTrue(groupNodeFlowFileActivity.getLatestActivityTime().isPresent()); + } + + @Test + public void testUpdateFlowFileActivityNoActivityTimeWhenFlowReportsNone() { + when(statelessFlow.getLatestActivityTime()).thenReturn(OptionalLong.empty()); + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.RECEIVE, 100L)); + + task.updateFlowFileActivity(statelessProvRepo); + + assertTrue(groupNodeFlowFileActivity.getLatestActivityTime().isEmpty()); + } + + @Test + public void testUpdateFlowFileActivityAccumulatesAcrossMultipleCalls() { + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.RECEIVE, 100L)); + task.updateFlowFileActivity(statelessProvRepo); + + statelessProvenanceEvents.clear(); + statelessProvenanceEvents.add(createProvenanceEvent(ProvenanceEventType.SEND, 50L)); + task.updateFlowFileActivity(statelessProvRepo); + + final FlowFileTransferCounts counts = groupNodeFlowFileActivity.getTransferCounts(); + assertEquals(1L, counts.getReceivedCount()); + assertEquals(100L, counts.getReceivedBytes()); + assertEquals(1L, counts.getSentCount()); + assertEquals(50L, counts.getSentBytes()); + } + + private ProvenanceEventRecord createProvenanceEvent(final ProvenanceEventType eventType, final long fileSize) { + final StandardProvenanceEventRecord.Builder builder = new StandardProvenanceEventRecord.Builder() + .setEventType(eventType) + .setComponentId("component-1") + .setEventTime(System.currentTimeMillis()) + .setFlowFileUUID("uuid-1") + .setComponentType("Unit Test") + .setCurrentContentClaim(null, null, null, null, fileSize); + + switch (eventType) { + case RECEIVE: + case SEND: + case FETCH: + builder.setTransitUri("http://localhost/test"); + break; + case ROUTE: + builder.setRelationship("success"); + break; + default: + break; + } + + return builder.build(); + } + private FlowFileRecord createFlowFile() { final ResourceClaim resourceClaim = new StandardResourceClaim(resourceClaimManager, "container", "section", "1", false); final ContentClaim contentClaim = new StandardContentClaim(resourceClaim, 0L); diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/manifest/StandardRuntimeManifestServiceTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/manifest/StandardRuntimeManifestServiceTest.java index fae3db4f85e7..fe9b3d93ef7c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/manifest/StandardRuntimeManifestServiceTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/manifest/StandardRuntimeManifestServiceTest.java @@ -31,6 +31,7 @@ import org.apache.nifi.nar.PythonBundle; import org.apache.nifi.processor.Processor; import org.apache.nifi.python.PythonProcessorDetails; +import org.apache.nifi.web.ResourceNotFoundException; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; @@ -38,11 +39,14 @@ import java.util.Arrays; import java.util.HashSet; import java.util.List; +import java.util.Map; import java.util.Set; import static java.util.Collections.emptySet; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -163,6 +167,50 @@ public void testGetPythonManifest() { assertEquals(0, controllerServiceDefinitions.size()); } + @Test + public void testDiscoverStepDocumentation() { + when(extensionManager.getBundle(testComponentsBundle.getBundleDetails().getCoordinate())).thenReturn(testComponentsBundle); + + final BundleCoordinate coordinate = testComponentsBundle.getBundleDetails().getCoordinate(); + final Map stepDocs = runtimeManifestService.discoverStepDocumentation( + coordinate.getGroup(), coordinate.getId(), coordinate.getVersion(), "org.example.TestConnector"); + + assertNotNull(stepDocs); + assertEquals(2, stepDocs.size()); + assertTrue(stepDocs.containsKey("Test Step")); + assertTrue(stepDocs.containsKey("Another Test Step")); + + final File testStepFile = stepDocs.get("Test Step"); + assertNotNull(testStepFile); + assertTrue(testStepFile.exists()); + assertEquals("Test_Step.md", testStepFile.getName()); + + final File anotherTestStepFile = stepDocs.get("Another Test Step"); + assertNotNull(anotherTestStepFile); + assertTrue(anotherTestStepFile.exists()); + assertEquals("Another_Test_Step.md", anotherTestStepFile.getName()); + } + + @Test + public void testDiscoverStepDocumentationWithNonExistentConnector() { + when(extensionManager.getBundle(testComponentsBundle.getBundleDetails().getCoordinate())).thenReturn(testComponentsBundle); + + final BundleCoordinate coordinate = testComponentsBundle.getBundleDetails().getCoordinate(); + final Map stepDocs = runtimeManifestService.discoverStepDocumentation( + coordinate.getGroup(), coordinate.getId(), coordinate.getVersion(), "org.example.NonExistentConnector"); + + assertNotNull(stepDocs); + assertTrue(stepDocs.isEmpty()); + } + + @Test + public void testDiscoverStepDocumentationWithNonExistentBundle() { + when(extensionManager.getBundle(new BundleCoordinate("org.example", "nonexistent", "1.0.0"))).thenReturn(null); + + assertThrows(ResourceNotFoundException.class, () -> + runtimeManifestService.discoverStepDocumentation("org.example", "nonexistent", "1.0.0", "org.example.TestConnector")); + } + /** * Override getFrameworkBundle to provide a mocked Bundle. */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactoryTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactoryTest.java new file mode 100644 index 000000000000..5692c82fd90b --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/provenance/StandardProvenanceAuthorizableFactoryTest.java @@ -0,0 +1,242 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.provenance; + +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.resource.DataAuthorizable; +import org.apache.nifi.authorization.resource.EnforcePolicyPermissionsThroughBaseResource; +import org.apache.nifi.authorization.resource.ProvenanceDataAuthorizable; +import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.groups.RemoteProcessGroup; +import org.apache.nifi.remote.RemoteGroupPort; +import org.apache.nifi.web.ResourceNotFoundException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class StandardProvenanceAuthorizableFactoryTest { + + private StandardProvenanceAuthorizableFactory factory; + + @Mock + private FlowController flowController; + + @Mock + private FlowManager flowManager; + + @Mock + private ProcessGroup rootGroup; + + @Mock + private Connection rootConnection; + + @Mock + private Connection connectorConnection; + + @Mock + private Connectable connectable; + + @Mock + private Connectable connectionSource; + + @Mock + private Connectable connectorConnectionSource; + + @Mock + private RemoteProcessGroup remoteProcessGroup; + + private static final String ROOT_GROUP_ID = "root-group-id"; + private static final String CONNECTABLE_ID = "connectable-id"; + private static final String ROOT_CONNECTION_ID = "root-connection-id"; + private static final String CONNECTOR_CONNECTION_ID = "connector-connection-id"; + private static final String ROOT_REMOTE_PORT_ID = "root-remote-port-id"; + private static final String CONNECTOR_REMOTE_PORT_ID = "connector-remote-port-id"; + private static final String NON_EXISTENT_ID = "non-existent-id"; + + @BeforeEach + void setUp() { + factory = new StandardProvenanceAuthorizableFactory(flowController); + + when(flowController.getFlowManager()).thenReturn(flowManager); + when(flowManager.getRootGroup()).thenReturn(rootGroup); + when(flowManager.getRootGroupId()).thenReturn(ROOT_GROUP_ID); + + // Connectable lookups (flat map, works for all components) + when(flowManager.findConnectable(CONNECTABLE_ID)).thenReturn(connectable); + when(flowManager.findConnectable(ROOT_CONNECTION_ID)).thenReturn(null); + when(flowManager.findConnectable(CONNECTOR_CONNECTION_ID)).thenReturn(null); + when(flowManager.findConnectable(NON_EXISTENT_ID)).thenReturn(null); + + // Connection lookups via FlowController (includes connector-managed PGs) + when(rootConnection.getSource()).thenReturn(connectionSource); + when(connectorConnection.getSource()).thenReturn(connectorConnectionSource); + when(flowController.findConnectionIncludingConnectorManaged(ROOT_CONNECTION_ID)).thenReturn(rootConnection); + when(flowController.findConnectionIncludingConnectorManaged(CONNECTOR_CONNECTION_ID)).thenReturn(connectorConnection); + when(flowController.findConnectionIncludingConnectorManaged(NON_EXISTENT_ID)).thenReturn(null); + + // Remote group port lookups via FlowController + when(flowController.findRemoteGroupPortIncludingConnectorManaged(NON_EXISTENT_ID)).thenReturn(null); + } + + @Nested + class CreateLocalDataAuthorizable { + + @Test + void testRootGroupIdReturnsDataAuthorizableForRootGroup() { + final Authorizable result = factory.createLocalDataAuthorizable(ROOT_GROUP_ID); + + assertInstanceOf(DataAuthorizable.class, result); + assertEquals(rootGroup, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testConnectableReturnsDataAuthorizableForConnectable() { + final Authorizable result = factory.createLocalDataAuthorizable(CONNECTABLE_ID); + + assertInstanceOf(DataAuthorizable.class, result); + assertEquals(connectable, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testConnectionInRootGroupReturnsDataAuthorizableForSource() { + final Authorizable result = factory.createLocalDataAuthorizable(ROOT_CONNECTION_ID); + + assertInstanceOf(DataAuthorizable.class, result); + assertEquals(connectionSource, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testConnectionInConnectorManagedGroupReturnsDataAuthorizableForSource() { + final Authorizable result = factory.createLocalDataAuthorizable(CONNECTOR_CONNECTION_ID); + + assertInstanceOf(DataAuthorizable.class, result); + assertEquals(connectorConnectionSource, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testNonExistentComponentThrowsResourceNotFoundException() { + assertThrows(ResourceNotFoundException.class, () -> + factory.createLocalDataAuthorizable(NON_EXISTENT_ID) + ); + } + } + + @Nested + class CreateProvenanceDataAuthorizable { + + @Test + void testRootGroupIdReturnsProvenanceDataAuthorizableForRootGroup() { + final Authorizable result = factory.createProvenanceDataAuthorizable(ROOT_GROUP_ID); + + assertInstanceOf(ProvenanceDataAuthorizable.class, result); + assertEquals(rootGroup, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testConnectableReturnsProvenanceDataAuthorizableForConnectable() { + final Authorizable result = factory.createProvenanceDataAuthorizable(CONNECTABLE_ID); + + assertInstanceOf(ProvenanceDataAuthorizable.class, result); + assertEquals(connectable, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testConnectionInRootGroupReturnsProvenanceDataAuthorizableForSource() { + final Authorizable result = factory.createProvenanceDataAuthorizable(ROOT_CONNECTION_ID); + + assertInstanceOf(ProvenanceDataAuthorizable.class, result); + assertEquals(connectionSource, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testConnectionInConnectorManagedGroupReturnsProvenanceDataAuthorizableForSource() { + final Authorizable result = factory.createProvenanceDataAuthorizable(CONNECTOR_CONNECTION_ID); + + assertInstanceOf(ProvenanceDataAuthorizable.class, result); + assertEquals(connectorConnectionSource, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testNonExistentComponentThrowsResourceNotFoundException() { + assertThrows(ResourceNotFoundException.class, () -> + factory.createProvenanceDataAuthorizable(NON_EXISTENT_ID) + ); + } + } + + @Nested + class CreateRemoteDataAuthorizable { + + private RemoteGroupPort rootRemoteGroupPort; + private RemoteGroupPort connectorRemoteGroupPort; + + @Mock + private RemoteProcessGroup connectorRemoteProcessGroup; + + @BeforeEach + void setUpRemotePorts() { + rootRemoteGroupPort = mock(RemoteGroupPort.class); + connectorRemoteGroupPort = mock(RemoteGroupPort.class); + + when(flowController.findRemoteGroupPortIncludingConnectorManaged(ROOT_REMOTE_PORT_ID)).thenReturn(rootRemoteGroupPort); + when(rootRemoteGroupPort.getRemoteProcessGroup()).thenReturn(remoteProcessGroup); + + when(flowController.findRemoteGroupPortIncludingConnectorManaged(CONNECTOR_REMOTE_PORT_ID)).thenReturn(connectorRemoteGroupPort); + when(connectorRemoteGroupPort.getRemoteProcessGroup()).thenReturn(connectorRemoteProcessGroup); + } + + @Test + void testRemoteGroupPortInRootGroupReturnsDataAuthorizableForRemoteProcessGroup() { + final Authorizable result = factory.createRemoteDataAuthorizable(ROOT_REMOTE_PORT_ID); + + assertInstanceOf(DataAuthorizable.class, result); + assertEquals(remoteProcessGroup, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testRemoteGroupPortInConnectorManagedGroupReturnsDataAuthorizableForRemoteProcessGroup() { + final Authorizable result = factory.createRemoteDataAuthorizable(CONNECTOR_REMOTE_PORT_ID); + + assertInstanceOf(DataAuthorizable.class, result); + assertEquals(connectorRemoteProcessGroup, ((EnforcePolicyPermissionsThroughBaseResource) result).getBaseAuthorizable()); + } + + @Test + void testNonExistentRemoteGroupPortThrowsResourceNotFoundException() { + assertThrows(ResourceNotFoundException.class, () -> + factory.createRemoteDataAuthorizable(NON_EXISTENT_ID) + ); + } + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/registry/flow/mapping/NiFiRegistryFlowMapperTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapperTest.java similarity index 99% rename from nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/registry/flow/mapping/NiFiRegistryFlowMapperTest.java rename to nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapperTest.java index 489db2e7c5e1..196fdc8cc646 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/registry/flow/mapping/NiFiRegistryFlowMapperTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapperTest.java @@ -119,7 +119,7 @@ @ExtendWith(MockitoExtension.class) @MockitoSettings(strictness = Strictness.LENIENT) -public class NiFiRegistryFlowMapperTest { +public class VersionedComponentFlowMapperTest { private static final String PARAMETER_PROVIDER_ID = "id"; @Mock @@ -135,7 +135,7 @@ public class NiFiRegistryFlowMapperTest { @Mock private ParameterProvider parameterProvider; - private final NiFiRegistryFlowMapper flowMapper = new NiFiRegistryFlowMapper(extensionManager); + private final VersionedComponentFlowMapper flowMapper = new VersionedComponentFlowMapper(extensionManager); private int counter = 1; diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.components.connector.Connector b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.components.connector.Connector new file mode 100644 index 000000000000..93fb310b0aca --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.components.connector.Connector @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.controller.flow.NopConnector +org.apache.nifi.components.connector.DynamicFlowConnector +org.apache.nifi.components.connector.DynamicAllowableValuesConnector diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.controller.ControllerService b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.controller.ControllerService index 373a1f8e1d22..d9e0a9a8385c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.controller.ControllerService +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.controller.ControllerService @@ -12,8 +12,11 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + org.apache.nifi.controller.service.util.TestControllerService org.apache.nifi.controller.service.mock.ServiceA org.apache.nifi.controller.service.mock.ServiceB org.apache.nifi.controller.service.mock.ServiceC -org.apache.nifi.controller.service.mock.ServiceD \ No newline at end of file +org.apache.nifi.controller.service.mock.ServiceD + +org.apache.nifi.components.connector.services.impl.StandardCounterService \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.processor.Processor b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.processor.Processor index 56740ff7c350..ff061c320029 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.processor.Processor +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/META-INF/services/org.apache.nifi.processor.Processor @@ -12,6 +12,15 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + org.apache.nifi.test.processors.ModifiesClasspathProcessor org.apache.nifi.test.processors.ModifiesClasspathNoAnnotationProcessor -org.apache.nifi.controller.service.mock.DummyProcessor \ No newline at end of file +org.apache.nifi.controller.service.mock.DummyProcessor +org.apache.nifi.components.connector.processors.CreateDummyFlowFile +org.apache.nifi.components.connector.processors.DuplicateFlowFile +org.apache.nifi.components.connector.processors.OverwriteFlowFile +org.apache.nifi.components.connector.processors.TerminateFlowFile +org.apache.nifi.components.connector.processors.LogFlowFileContents +org.apache.nifi.components.connector.processors.ExposeFileValues +org.apache.nifi.components.connector.processors.Sleep +org.apache.nifi.components.connector.processors.OnPropertyModifiedTracker \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Another_Test_Step.md b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Another_Test_Step.md new file mode 100644 index 000000000000..4673395a0c94 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Another_Test_Step.md @@ -0,0 +1,4 @@ +# Another Test Step + +Documentation for another step. + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Test_Step.md b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Test_Step.md new file mode 100644 index 000000000000..22a6c822b828 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/TestRuntimeManifest/nifi-test-components-nar/META-INF/docs/steps/org.example.TestConnector/Test_Step.md @@ -0,0 +1,8 @@ +# Test Step Documentation + +This is test documentation for the Test Step. + +## Configuration + +Configure the test step properties as needed. + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/colors.txt b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/colors.txt new file mode 100644 index 000000000000..1fedccaeb90b --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/colors.txt @@ -0,0 +1,3 @@ +red +green +blue \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/choose-color.json b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/choose-color.json new file mode 100644 index 000000000000..cd29c728565c --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/choose-color.json @@ -0,0 +1,76 @@ +{ + "flowContents": { + "identifier": "9caa5337-2e56-3711-a344-b29bce96d3e2", + "instanceIdentifier": "77e61da1-0199-1000-ffff-ffff90f30b8d", + "name": "Choose Color", + "comments": "", + "position": { + "x": 61.0, + "y": -181.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "2abf1573-d8ee-38ad-a745-083fdb1a8903", + "instanceIdentifier": "09b902c7-c4b7-3e92-8246-ef0a0bb685e7", + "name": "Choose Color", + "comments": "", + "position": { + "x": -355.5, + "y": -399.0 + }, + "type": "org.apache.nifi.components.connector.processors.ExposeFileValues", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + }, + "propertyDescriptors": { + }, + "style": {}, + "schedulingPeriod": "1 min", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "9caa5337-2e56-3711-a344-b29bce96d3e2" + } + ], + "inputPorts": [], + "outputPorts": [], + "connections": [ + ], + "labels": [], + "funnels": [], + "controllerServices": [], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP" + }, + "externalControllerServices": {}, + "parameterContexts": { + }, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/generate-and-log-with-parameter.json b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/generate-and-log-with-parameter.json new file mode 100644 index 000000000000..5b46489242e5 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/generate-and-log-with-parameter.json @@ -0,0 +1,253 @@ +{ + "flowContents": { + "identifier": "9caa5337-2e56-3711-a344-b29bce96d3e2", + "instanceIdentifier": "77e61da1-0199-1000-ffff-ffff90f30b8d", + "name": "Generate and Log with Parameter", + "comments": "", + "position": { + "x": 61.0, + "y": -181.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "2abf1573-d8ee-38ad-a745-083fdb1a8903", + "instanceIdentifier": "09b902c7-c4b7-3e92-8246-ef0a0bb685e7", + "name": "CreateDummyFlowFile", + "comments": "", + "position": { + "x": -355.5, + "y": -399.0 + }, + "type": "org.apache.nifi.components.connector.processors.CreateDummyFlowFile", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + "Text": "#{Text}" + }, + "propertyDescriptors": { + "Text": { + "name": "Text", + "displayName": "Text", + "identifiesControllerService": false, + "sensitive": true, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "1 min", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "9caa5337-2e56-3711-a344-b29bce96d3e2" + }, + { + "identifier": "efefbde7-a9e8-3864-b1d2-9649981b39d1", + "instanceIdentifier": "77e69c78-0199-1000-0000-00001e16d8fc", + "name": "Sleep", + "comments": "", + "position": { + "x": -352.0, + "y": -152.0 + }, + "type": "org.apache.nifi.components.connector.processors.Sleep", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + "Sleep Duration": "#{Sleep Duration}", + "Sleep On Stopped": "true" + }, + "propertyDescriptors": { + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 25, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "9caa5337-2e56-3711-a344-b29bce96d3e2" + }, + { + "identifier": "3d651016-1570-37b6-b3a3-6575d1545cb7", + "instanceIdentifier": "81bb2507-6874-3bb7-a280-a327515aa5fe", + "name": "LogFlowFileContents", + "comments": "", + "position": { + "x": -352.0, + "y": 104.0 + }, + "type": "org.apache.nifi.components.connector.processors.LogFlowFileContents", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": {}, + "propertyDescriptors": {}, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 25, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [ + "success" + ], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "9caa5337-2e56-3711-a344-b29bce96d3e2" + } + ], + "inputPorts": [], + "outputPorts": [], + "connections": [ + { + "identifier": "17cbec18-38a6-36f6-ba7a-b5cc3775ef52", + "instanceIdentifier": "d303de27-6bb1-39c3-968b-64faabb17d27", + "name": "", + "source": { + "id": "2abf1573-d8ee-38ad-a745-083fdb1a8903", + "type": "PROCESSOR", + "groupId": "9caa5337-2e56-3711-a344-b29bce96d3e2", + "name": "CreateDummyFlowFile", + "comments": "", + "instanceIdentifier": "09b902c7-c4b7-3e92-8246-ef0a0bb685e7" + }, + "destination": { + "id": "efefbde7-a9e8-3864-b1d2-9649981b39d1", + "type": "PROCESSOR", + "groupId": "9caa5337-2e56-3711-a344-b29bce96d3e2", + "name": "Sleep", + "comments": "", + "instanceIdentifier": "77e69c78-0199-1000-0000-00001e16d8fc" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "9caa5337-2e56-3711-a344-b29bce96d3e2" + }, + { + "identifier": "5b5e23d7-e6ce-3122-bfe7-3c7bb194893b", + "instanceIdentifier": "77e6bb8c-0199-1000-ffff-ffffe70db445", + "name": "", + "source": { + "id": "efefbde7-a9e8-3864-b1d2-9649981b39d1", + "type": "PROCESSOR", + "groupId": "9caa5337-2e56-3711-a344-b29bce96d3e2", + "name": "Sleep", + "comments": "", + "instanceIdentifier": "77e69c78-0199-1000-0000-00001e16d8fc" + }, + "destination": { + "id": "3d651016-1570-37b6-b3a3-6575d1545cb7", + "type": "PROCESSOR", + "groupId": "9caa5337-2e56-3711-a344-b29bce96d3e2", + "name": "LogFlowFileContents", + "comments": "", + "instanceIdentifier": "81bb2507-6874-3bb7-a280-a327515aa5fe" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "9caa5337-2e56-3711-a344-b29bce96d3e2" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [], + "parameterContextName": "Generate and Log", + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP" + }, + "externalControllerServices": {}, + "parameterContexts": { + "Generate and Log": { + "name": "Generate and Log", + "parameters": [ + { + "name": "Text", + "description": "", + "sensitive": false, + "provided": false, + "value": "Hello" + }, { + "name": "Sleep Duration", + "description": "", + "sensitive": false, + "provided": false, + "value": "5 sec" + } + ], + "inheritedParameterContexts": [], + "componentType": "PARAMETER_CONTEXT" + } + }, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/generate-duplicate-log-flow.json b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/generate-duplicate-log-flow.json new file mode 100644 index 000000000000..b39552388e8d --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/generate-duplicate-log-flow.json @@ -0,0 +1,690 @@ +{ + "flowContents": { + "identifier": "55129302-50a3-3d2f-8abd-b2479f4e9be3", + "instanceIdentifier": "544710c8-0199-1000-0000-00002aba2366", + "name": "Connector", + "comments": "", + "position": { + "x": 33.0, + "y": -170.0 + }, + "processGroups": [ + { + "identifier": "ce570e99-1208-3f71-0000-00006b7e3901", + "instanceIdentifier": "8f439d4b-fc9a-3f25-96ce-9682838c06b0", + "name": "Duplicates", + "comments": "", + "position": { + "x": -192.0, + "y": -24.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "ac87c05c-fdeb-32bc-8bef-304c401331ca", + "instanceIdentifier": "5448f92d-0199-1000-0000-00005958156f", + "name": "Create Duplicates", + "comments": "", + "position": { + "x": -200.0, + "y": -152.0 + }, + "type": "org.apache.nifi.components.connector.processors.DuplicateFlowFile", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + "Number of Duplicates": "3" + }, + "propertyDescriptors": { + "Number of Copies": { + "name": "Number of Duplicates", + "displayName": "Number of Duplicates", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + } + ], + "inputPorts": [ + { + "identifier": "b94406cc-5951-398f-84eb-db602ec72a4a", + "instanceIdentifier": "5447b963-0199-1000-ffff-ffffd01a6d32", + "name": "In", + "position": { + "x": -144.0, + "y": -325.0 + }, + "type": "INPUT_PORT", + "concurrentlySchedulableTaskCount": 1, + "scheduledState": "ENABLED", + "allowRemoteAccess": false, + "portFunction": "STANDARD", + "componentType": "INPUT_PORT", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + } + ], + "outputPorts": [ + { + "identifier": "3aebcf02-2c3e-38aa-ab15-61c2481c3e0a", + "instanceIdentifier": "5447c88a-0199-1000-0000-00002a2f67b2", + "name": "Out", + "position": { + "x": -144.0, + "y": 104.0 + }, + "type": "OUTPUT_PORT", + "concurrentlySchedulableTaskCount": 1, + "scheduledState": "ENABLED", + "allowRemoteAccess": false, + "portFunction": "STANDARD", + "componentType": "OUTPUT_PORT", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + } + ], + "connections": [ + { + "identifier": "2039d770-d9ab-3e71-ae03-569214755a46", + "instanceIdentifier": "5449dab5-0199-1000-0000-000011c5a49a", + "name": "3", + "source": { + "id": "ac87c05c-fdeb-32bc-8bef-304c401331ca", + "type": "PROCESSOR", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Create Duplicates", + "comments": "", + "instanceIdentifier": "5448f92d-0199-1000-0000-00005958156f" + }, + "destination": { + "id": "3aebcf02-2c3e-38aa-ab15-61c2481c3e0a", + "type": "OUTPUT_PORT", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Out", + "instanceIdentifier": "5447c88a-0199-1000-0000-00002a2f67b2" + }, + "labelIndex": 0, + "zIndex": 4, + "selectedRelationships": [ + "3" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [ + { + "x": 226.0, + "y": 20.0 + } + ], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + }, + { + "identifier": "2bdfebb9-9698-3104-95a7-acd670ecb355", + "instanceIdentifier": "54491de8-0199-1000-ffff-ffffa4414ae3", + "name": "", + "source": { + "id": "b94406cc-5951-398f-84eb-db602ec72a4a", + "type": "INPUT_PORT", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "In", + "instanceIdentifier": "5447b963-0199-1000-ffff-ffffd01a6d32" + }, + "destination": { + "id": "ac87c05c-fdeb-32bc-8bef-304c401331ca", + "type": "PROCESSOR", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Create Duplicates", + "comments": "", + "instanceIdentifier": "5448f92d-0199-1000-0000-00005958156f" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + }, + { + "identifier": "2361063e-00eb-3090-8cee-650d9bbe340c", + "instanceIdentifier": "5449aaf6-0199-1000-ffff-ffffbf429121", + "name": "1", + "source": { + "id": "ac87c05c-fdeb-32bc-8bef-304c401331ca", + "type": "PROCESSOR", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Create Duplicates", + "comments": "", + "instanceIdentifier": "5448f92d-0199-1000-0000-00005958156f" + }, + "destination": { + "id": "3aebcf02-2c3e-38aa-ab15-61c2481c3e0a", + "type": "OUTPUT_PORT", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Out", + "instanceIdentifier": "5447c88a-0199-1000-0000-00002a2f67b2" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "1" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + }, + { + "identifier": "4cecc7a5-8156-328e-b18f-54c612924b9c", + "instanceIdentifier": "5449c2b5-0199-1000-0000-000076b2bd73", + "name": "2", + "source": { + "id": "ac87c05c-fdeb-32bc-8bef-304c401331ca", + "type": "PROCESSOR", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Create Duplicates", + "comments": "", + "instanceIdentifier": "5448f92d-0199-1000-0000-00005958156f" + }, + "destination": { + "id": "3aebcf02-2c3e-38aa-ab15-61c2481c3e0a", + "type": "OUTPUT_PORT", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Out", + "instanceIdentifier": "5447c88a-0199-1000-0000-00002a2f67b2" + }, + "labelIndex": 0, + "zIndex": 3, + "selectedRelationships": [ + "2" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [ + { + "x": -274.0, + "y": 20.0 + } + ], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "ce570e99-1208-3f71-0000-00006b7e3901" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP", + "groupIdentifier": "55129302-50a3-3d2f-8abd-b2479f4e9be3" + }, + { + "identifier": "e031373e-71a2-3d7f-0000-00006cf360cf", + "instanceIdentifier": "a8a23ec0-3279-3172-939b-adb5bdaf59e1", + "name": "Destination", + "comments": "", + "position": { + "x": -192.0, + "y": 264.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "43814190-1b7e-3f38-8844-10c97f1d6b95", + "instanceIdentifier": "544a0c77-0199-1000-ffff-ffffd81dd5f1", + "name": "Terminate FlowFile", + "comments": "", + "position": { + "x": -216.0, + "y": -64.0 + }, + "type": "org.apache.nifi.components.connector.processors.TerminateFlowFile", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + }, + "propertyDescriptors": { + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 25, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "e031373e-71a2-3d7f-0000-00006cf360cf" + } + ], + "inputPorts": [ + { + "identifier": "5daa9ba5-dec6-397d-8cd5-b3f624bda832", + "instanceIdentifier": "5447ea94-0199-1000-0000-00006ed76b5a", + "name": "In", + "position": { + "x": -160.0, + "y": -232.0 + }, + "type": "INPUT_PORT", + "concurrentlySchedulableTaskCount": 1, + "scheduledState": "ENABLED", + "allowRemoteAccess": false, + "portFunction": "STANDARD", + "componentType": "INPUT_PORT", + "groupIdentifier": "e031373e-71a2-3d7f-0000-00006cf360cf" + } + ], + "outputPorts": [], + "connections": [ + { + "identifier": "1fbf6c9c-362f-3906-a3b5-0761874b5791", + "instanceIdentifier": "544a3fb0-0199-1000-0000-000033504ad9", + "name": "", + "source": { + "id": "5daa9ba5-dec6-397d-8cd5-b3f624bda832", + "type": "INPUT_PORT", + "groupId": "e031373e-71a2-3d7f-0000-00006cf360cf", + "name": "In", + "instanceIdentifier": "5447ea94-0199-1000-0000-00006ed76b5a" + }, + "destination": { + "id": "43814190-1b7e-3f38-8844-10c97f1d6b95", + "type": "PROCESSOR", + "groupId": "e031373e-71a2-3d7f-0000-00006cf360cf", + "name": "Terminate FlowFile", + "comments": "", + "instanceIdentifier": "544a0c77-0199-1000-ffff-ffffd81dd5f1" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "e031373e-71a2-3d7f-0000-00006cf360cf" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP", + "groupIdentifier": "55129302-50a3-3d2f-8abd-b2479f4e9be3" + }, + { + "identifier": "06847e96-c5c7-35fe-88fc-64085f82192f", + "instanceIdentifier": "54472440-0199-1000-0000-00000b009f05", + "name": "Source", + "comments": "", + "position": { + "x": -192.0, + "y": -304.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "ea98a6b9-3074-33a4-9d58-903767e34a7a", + "instanceIdentifier": "5448a4ad-0199-1000-0000-000078c68790", + "name": "Set Contents", + "comments": "", + "position": { + "x": -120.0, + "y": -56.0 + }, + "type": "org.apache.nifi.components.connector.processors.OverwriteFlowFile", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + "Content": "Hi there!" + }, + "propertyDescriptors": { + "Content": { + "name": "Content", + "displayName": "Content", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "0 sec", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 25, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "06847e96-c5c7-35fe-88fc-64085f82192f" + }, + { + "identifier": "53e0f8b5-e04c-34c1-906a-4a5895fa8256", + "instanceIdentifier": "5448581a-0199-1000-ffff-ffff814d4715", + "name": "Create FlowFile", + "comments": "", + "position": { + "x": -120.0, + "y": -280.0 + }, + "type": "org.apache.nifi.components.connector.processors.CreateDummyFlowFile", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + }, + "propertyDescriptors": { + }, + "style": {}, + "schedulingPeriod": "1 min", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": [], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "06847e96-c5c7-35fe-88fc-64085f82192f" + } + ], + "inputPorts": [], + "outputPorts": [ + { + "identifier": "54ed1d21-458e-38d9-9155-53a7b959ed8e", + "instanceIdentifier": "5447a2fc-0199-1000-ffff-ffffc38ab2be", + "name": "Out", + "position": { + "x": -64.0, + "y": 184.0 + }, + "type": "OUTPUT_PORT", + "concurrentlySchedulableTaskCount": 1, + "scheduledState": "ENABLED", + "allowRemoteAccess": false, + "portFunction": "STANDARD", + "componentType": "OUTPUT_PORT", + "groupIdentifier": "06847e96-c5c7-35fe-88fc-64085f82192f" + } + ], + "connections": [ + { + "identifier": "9161e363-0085-361a-8d28-2107c3f6f8fd", + "instanceIdentifier": "5448bbc2-0199-1000-ffff-ffffbb98a878", + "name": "", + "source": { + "id": "53e0f8b5-e04c-34c1-906a-4a5895fa8256", + "type": "PROCESSOR", + "groupId": "06847e96-c5c7-35fe-88fc-64085f82192f", + "name": "Create FlowFile", + "comments": "", + "instanceIdentifier": "5448581a-0199-1000-ffff-ffff814d4715" + }, + "destination": { + "id": "ea98a6b9-3074-33a4-9d58-903767e34a7a", + "type": "PROCESSOR", + "groupId": "06847e96-c5c7-35fe-88fc-64085f82192f", + "name": "Set Contents", + "comments": "", + "instanceIdentifier": "5448a4ad-0199-1000-0000-000078c68790" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "06847e96-c5c7-35fe-88fc-64085f82192f" + }, + { + "identifier": "b2d43983-2f9a-3ffa-bb70-d633a8c5b1c8", + "instanceIdentifier": "5448c84d-0199-1000-0000-00003f3bdee2", + "name": "", + "source": { + "id": "ea98a6b9-3074-33a4-9d58-903767e34a7a", + "type": "PROCESSOR", + "groupId": "06847e96-c5c7-35fe-88fc-64085f82192f", + "name": "Set Contents", + "comments": "", + "instanceIdentifier": "5448a4ad-0199-1000-0000-000078c68790" + }, + "destination": { + "id": "54ed1d21-458e-38d9-9155-53a7b959ed8e", + "type": "OUTPUT_PORT", + "groupId": "06847e96-c5c7-35fe-88fc-64085f82192f", + "name": "Out", + "instanceIdentifier": "5447a2fc-0199-1000-ffff-ffffc38ab2be" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "success" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "06847e96-c5c7-35fe-88fc-64085f82192f" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP", + "groupIdentifier": "55129302-50a3-3d2f-8abd-b2479f4e9be3" + } + ], + "remoteProcessGroups": [], + "processors": [], + "inputPorts": [], + "outputPorts": [], + "connections": [ + { + "identifier": "15b78f0d-0c72-3e45-95b7-4441f8274b24", + "instanceIdentifier": "54481ee4-0199-1000-ffff-ffff9110fe5e", + "name": "", + "source": { + "id": "3aebcf02-2c3e-38aa-ab15-61c2481c3e0a", + "type": "OUTPUT_PORT", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "Out", + "instanceIdentifier": "5447c88a-0199-1000-0000-00002a2f67b2" + }, + "destination": { + "id": "5daa9ba5-dec6-397d-8cd5-b3f624bda832", + "type": "INPUT_PORT", + "groupId": "e031373e-71a2-3d7f-0000-00006cf360cf", + "name": "In", + "instanceIdentifier": "5447ea94-0199-1000-0000-00006ed76b5a" + }, + "labelIndex": 0, + "zIndex": 2, + "selectedRelationships": [ + "" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "55129302-50a3-3d2f-8abd-b2479f4e9be3" + }, + { + "identifier": "9a522be3-0de3-328c-af6c-33f0204bc40e", + "instanceIdentifier": "5448016d-0199-1000-0000-00002669c0b1", + "name": "", + "source": { + "id": "54ed1d21-458e-38d9-9155-53a7b959ed8e", + "type": "OUTPUT_PORT", + "groupId": "06847e96-c5c7-35fe-88fc-64085f82192f", + "name": "Out", + "instanceIdentifier": "5447a2fc-0199-1000-ffff-ffffc38ab2be" + }, + "destination": { + "id": "b94406cc-5951-398f-84eb-db602ec72a4a", + "type": "INPUT_PORT", + "groupId": "ce570e99-1208-3f71-0000-00006b7e3901", + "name": "In", + "instanceIdentifier": "5447b963-0199-1000-ffff-ffffd01a6d32" + }, + "labelIndex": 0, + "zIndex": 1, + "selectedRelationships": [ + "" + ], + "backPressureObjectThreshold": 10000, + "backPressureDataSizeThreshold": "1 GB", + "flowFileExpiration": "0 sec", + "prioritizers": [], + "bends": [], + "loadBalanceStrategy": "DO_NOT_LOAD_BALANCE", + "partitioningAttribute": "", + "loadBalanceCompression": "DO_NOT_COMPRESS", + "componentType": "CONNECTION", + "groupIdentifier": "55129302-50a3-3d2f-8abd-b2479f4e9be3" + } + ], + "labels": [], + "funnels": [], + "controllerServices": [], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP" + }, + "externalControllerServices": {}, + "parameterContexts": {}, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} \ No newline at end of file diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/on-property-modified-tracker.json b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/on-property-modified-tracker.json new file mode 100644 index 000000000000..1f4f11c0fb39 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/flows/on-property-modified-tracker.json @@ -0,0 +1,98 @@ +{ + "flowContents": { + "identifier": "on-property-modified-root", + "instanceIdentifier": "on-property-modified-instance", + "name": "OnPropertyModified Test Flow", + "comments": "", + "position": { + "x": 0.0, + "y": 0.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [ + { + "identifier": "on-property-modified-tracker", + "instanceIdentifier": "on-property-modified-tracker-instance", + "name": "OnPropertyModifiedTracker", + "comments": "", + "position": { + "x": 0.0, + "y": 0.0 + }, + "type": "org.apache.nifi.components.connector.processors.OnPropertyModifiedTracker", + "bundle": { + "group": "default", + "artifact": "system", + "version": "unversioned" + }, + "properties": { + "Configured Number": "#{CONFIGURED_NUMBER}" + }, + "propertyDescriptors": { + "Configured Number": { + "name": "Configured Number", + "displayName": "Configured Number", + "identifiesControllerService": false, + "sensitive": false, + "dynamic": false + } + }, + "style": {}, + "schedulingPeriod": "1 min", + "schedulingStrategy": "TIMER_DRIVEN", + "executionNode": "ALL", + "penaltyDuration": "30 sec", + "yieldDuration": "1 sec", + "bulletinLevel": "WARN", + "runDurationMillis": 0, + "concurrentlySchedulableTaskCount": 1, + "autoTerminatedRelationships": ["success"], + "scheduledState": "ENABLED", + "retryCount": 10, + "retriedRelationships": [], + "backoffMechanism": "PENALIZE_FLOWFILE", + "maxBackoffPeriod": "10 mins", + "componentType": "PROCESSOR", + "groupIdentifier": "on-property-modified-root" + } + ], + "inputPorts": [], + "outputPorts": [], + "connections": [], + "labels": [], + "funnels": [], + "controllerServices": [], + "parameterContextName": "OnPropertyModifiedContext", + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "scheduledState": "ENABLED", + "executionEngine": "INHERITED", + "maxConcurrentTasks": 1, + "statelessFlowTimeout": "1 min", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE", + "componentType": "PROCESS_GROUP" + }, + "externalControllerServices": {}, + "parameterContexts": { + "OnPropertyModifiedContext": { + "name": "OnPropertyModifiedContext", + "parameters": [ + { + "name": "CONFIGURED_NUMBER", + "description": "", + "sensitive": false, + "provided": false, + "value": null + } + ], + "inheritedParameterContexts": [], + "componentType": "PARAMETER_CONTEXT" + } + }, + "flowEncodingVersion": "1.0", + "parameterProviders": {}, + "latest": false +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/StandardExtensionDiscoveringManager.java b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/StandardExtensionDiscoveringManager.java index 5a1d3dee5202..aa47236cf173 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/StandardExtensionDiscoveringManager.java +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/StandardExtensionDiscoveringManager.java @@ -29,6 +29,7 @@ import org.apache.nifi.components.ClassloaderIsolationKeyProvider; import org.apache.nifi.components.ConfigurableComponent; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.Connector; import org.apache.nifi.components.state.StateProvider; import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.leader.election.LeaderElectionManager; @@ -136,6 +137,7 @@ public StandardExtensionDiscoveringManager(final Collection()); definitionMap.put(FlowActionReporter.class, new HashSet<>()); definitionMap.put(ComponentMetricReporter.class, new HashSet<>()); + definitionMap.put(Connector.class, new HashSet<>()); additionalExtensionTypes.forEach(type -> definitionMap.putIfAbsent(type, new HashSet<>())); } @@ -408,9 +410,12 @@ private Set getServiceFileURLs(final Bundle bundle, final Class extensio serviceResourceUrls.add(serviceResourceUrlEnum.nextElement()); } - final Enumeration parentResourceUrlEnum = bundle.getClassLoader().getParent().getResources(servicesFile); - while (parentResourceUrlEnum.hasMoreElements()) { - serviceResourceUrls.remove(parentResourceUrlEnum.nextElement()); + final ClassLoader parentClassLoader = bundle.getClassLoader().getParent(); + if (parentClassLoader != null) { + final Enumeration parentResourceUrlEnum = parentClassLoader.getResources(servicesFile); + while (parentResourceUrlEnum.hasMoreElements()) { + serviceResourceUrls.remove(parentResourceUrlEnum.nextElement()); + } } return serviceResourceUrls; @@ -421,7 +426,7 @@ protected void loadExtension(final String extensionClassName, final Class ext } protected void registerExtensionClass(final Class extensionType, final String implementationClassName, final Bundle bundle) { - final Set registeredClasses = definitionMap.get(extensionType); + final Set registeredClasses = definitionMap.computeIfAbsent(extensionType, type -> new HashSet<>()); registerServiceClass(implementationClassName, extensionType, classNameBundleLookup, bundleCoordinateClassesLookup, bundle, registeredClasses); } @@ -507,7 +512,8 @@ public Class getClass(final ExtensionDefinition extensionDefinition) { */ private static boolean multipleVersionsAllowed(Class type) { return Processor.class.isAssignableFrom(type) || ControllerService.class.isAssignableFrom(type) || ReportingTask.class.isAssignableFrom(type) - || FlowAnalysisRule.class.isAssignableFrom(type) || ParameterProvider.class.isAssignableFrom(type) || FlowRegistryClient.class.isAssignableFrom(type); + || FlowAnalysisRule.class.isAssignableFrom(type) || ParameterProvider.class.isAssignableFrom(type) || FlowRegistryClient.class.isAssignableFrom(type) + || Connector.class.isAssignableFrom(type); } protected boolean isInstanceClassLoaderRequired(final String classType, final Bundle bundle) { @@ -859,6 +865,10 @@ public synchronized ConfigurableComponent getTempComponent(final String classTyp tempComponent = pythonBridge.createProcessor(procId, classType, bundleCoordinate.getVersion(), false, false); } else { final Class componentClass = Class.forName(classType, true, bundleClassLoader); + if (!ConfigurableComponent.class.isAssignableFrom(componentClass)) { + return null; + } + tempComponent = (ConfigurableComponent) componentClass.getDeclaredConstructor().newInstance(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/java/org/apache/nifi/nar/DummyConnector.java b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/java/org/apache/nifi/nar/DummyConnector.java new file mode 100644 index 000000000000..2a2b21cde1b3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/java/org/apache/nifi/nar/DummyConnector.java @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.nar; + +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.ConnectorInitializationContext; +import org.apache.nifi.components.connector.ConnectorValidationContext; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +@Tags({"test", "connector"}) +public class DummyConnector implements Connector { + private ConnectorInitializationContext context; + + public ConnectorInitializationContext getContext() { + return context; + } + + @Override + public void initialize(final ConnectorInitializationContext connectorInitializationContext) { + this.context = connectorInitializationContext; + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return null; + } + + @Override + public void start(final FlowContext flowContext) throws FlowUpdateException { + + } + + @Override + public void stop(final FlowContext flowContext) throws FlowUpdateException { + + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void onConfigurationStepConfigured(final String stepName, final FlowContext flowContext) { + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) { + } + + @Override + public void abortUpdate(final FlowContext flowContext, final Throwable cause) { + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List verify(final FlowContext flowContext) { + return List.of(); + } + + @Override + public List validate(final FlowContext flowContext, final ConnectorValidationContext validationContext) { + return List.of(); + } + + @Override + public List validateConfigurationStep(final ConfigurationStep configurationStep, final ConnectorConfigurationContext connectorConfigurationContext, + final ConnectorValidationContext connectorValidationContext) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List fetchAllowableValues(final String stepName, final String propertyName, final FlowContext flowContext, final String filter) { + return List.of(); + } + + @Override + public CompletableFuture drainFlowFiles(final FlowContext flowContext) { + return CompletableFuture.completedFuture(null); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/java/org/apache/nifi/nar/StandardExtensionDiscoveringManagerConnectorTest.java b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/java/org/apache/nifi/nar/StandardExtensionDiscoveringManagerConnectorTest.java new file mode 100644 index 000000000000..7d2213ddb3b7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/java/org/apache/nifi/nar/StandardExtensionDiscoveringManagerConnectorTest.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.nar; + +import org.apache.nifi.bundle.Bundle; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.util.NiFiProperties; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +public class StandardExtensionDiscoveringManagerConnectorTest { + + @Test + public void testConnectorDiscoveryFromServiceFile() { + final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties("src/test/resources/nifi.properties"); + final Bundle systemBundle = SystemBundle.create(properties); + + final StandardExtensionDiscoveringManager manager = new StandardExtensionDiscoveringManager(); + manager.discoverExtensions(systemBundle, Collections.emptySet()); + + final Set connectorDefs = manager.getExtensions(Connector.class); + final ExtensionDefinition connectorDefinition = connectorDefs.stream() + .filter(def -> def.getImplementationClassName().equals("org.apache.nifi.nar.DummyConnector")) + .findFirst().orElseThrow(() -> new AssertionError("Did not find expected DummyConnector")); + + assertEquals(systemBundle, connectorDefinition.getBundle()); + assertEquals(Connector.class, connectorDefinition.getExtensionType()); + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/resources/META-INF/services/org.apache.nifi.components.connector.Connector b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/resources/META-INF/services/org.apache.nifi.components.connector.Connector new file mode 100644 index 000000000000..89a28c784f16 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/test/resources/META-INF/services/org.apache.nifi.components.connector.Connector @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +org.apache.nifi.nar.NarThreadContextClassLoaderTest$WithPropertiesConstructor +org.apache.nifi.nar.NarThreadContextClassLoaderTest$WithDefaultConstructor + +org.apache.nifi.nar.DummyConnector diff --git a/nifi-framework-bundle/nifi-framework/nifi-headless-server/src/main/java/org/apache/nifi/headless/HeadlessNiFiServer.java b/nifi-framework-bundle/nifi-framework/nifi-headless-server/src/main/java/org/apache/nifi/headless/HeadlessNiFiServer.java index a2936c840f79..e59ea12d44c8 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-headless-server/src/main/java/org/apache/nifi/headless/HeadlessNiFiServer.java +++ b/nifi-framework-bundle/nifi-framework/nifi-headless-server/src/main/java/org/apache/nifi/headless/HeadlessNiFiServer.java @@ -29,6 +29,7 @@ import org.apache.nifi.bundle.Bundle; import org.apache.nifi.cluster.ClusterDetailsFactory; import org.apache.nifi.cluster.ConnectionState; +import org.apache.nifi.components.connector.StandaloneConnectorRequestReplicator; import org.apache.nifi.components.state.StateManagerProvider; import org.apache.nifi.controller.DecommissionTask; import org.apache.nifi.controller.FlowController; @@ -150,16 +151,18 @@ public void preDestruction() throws AuthorizerDestructionException { extensionManager, statusHistoryRepository, null, - stateManagerProvider + stateManagerProvider, + new StandaloneConnectorRequestReplicator() ); flowService = StandardFlowService.createStandaloneInstance( - flowController, - props, - null, // revision manager - null, // NAR Manager - null, // Asset Synchronizer - authorizer); + flowController, + props, + null, // revision manager + null, // NAR Manager + null, // Parameter Context Asset Synchronizer + null, // Connector Asset Synchronizer + authorizer); diagnosticsFactory = new BootstrapDiagnosticsFactory(); ((BootstrapDiagnosticsFactory) diagnosticsFactory).setFlowController(flowController); diff --git a/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java b/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java index 773ad44d674a..2399c295d632 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java +++ b/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java @@ -545,8 +545,9 @@ private static ExtensionMapping getJarExtensionMapping(final BundleCoordinate co final JarEntry controllerServiceEntry = jarFile.getJarEntry("META-INF/services/org.apache.nifi.controller.ControllerService"); final JarEntry parameterProviderEntry = jarFile.getJarEntry("META-INF/services/org.apache.nifi.parameter.ParameterProvider"); final JarEntry flowRegistryClientEntry = jarFile.getJarEntry("META-INF/services/org.apache.nifi.registry.flow.FlowRegistryClient"); + final JarEntry connectorEntry = jarFile.getJarEntry("META-INF/services/org.apache.nifi.components.connector.Connector"); - if (processorEntry == null && reportingTaskEntry == null && flowAnalysisRuleEntry == null && controllerServiceEntry == null && parameterProviderEntry == null) { + if (processorEntry == null && reportingTaskEntry == null && flowAnalysisRuleEntry == null && controllerServiceEntry == null && parameterProviderEntry == null && connectorEntry == null) { return mapping; } @@ -556,6 +557,7 @@ private static ExtensionMapping getJarExtensionMapping(final BundleCoordinate co mapping.addAllControllerServices(coordinate, detectNiFiComponents(jarFile, controllerServiceEntry)); mapping.addAllParameterProviders(coordinate, detectNiFiComponents(jarFile, parameterProviderEntry)); mapping.addAllFlowRegistryClients(coordinate, detectNiFiComponents(jarFile, flowRegistryClientEntry)); + mapping.addAllConnectors(coordinate, detectNiFiComponents(jarFile, connectorEntry)); return mapping; } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-repository-models/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecord.java b/nifi-framework-bundle/nifi-framework/nifi-repository-models/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecord.java index 966c7138711a..efd2d3d69390 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-repository-models/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecord.java +++ b/nifi-framework-bundle/nifi-framework/nifi-repository-models/src/main/java/org/apache/nifi/controller/repository/StandardRepositoryRecord.java @@ -36,6 +36,7 @@ public class StandardRepositoryRecord implements RepositoryRecord { private final FlowFileRecord originalFlowFileRecord; private final FlowFileQueue originalQueue; private String swapLocation; + private String originalSwapLocation; private final Map originalAttributes; private Map updatedAttributes = null; private List transientClaims; @@ -103,11 +104,24 @@ public String getSwapLocation() { return swapLocation; } - public void setSwapLocation(final String swapLocation) { - this.swapLocation = swapLocation; - if (type != RepositoryRecordType.SWAP_OUT) { - setType(RepositoryRecordType.SWAP_IN); // we are swapping in a new record + public void setSwapLocation(final String swapLocation, final RepositoryRecordType swapType) { + if (swapType != RepositoryRecordType.SWAP_IN && swapType != RepositoryRecordType.SWAP_OUT + && swapType != RepositoryRecordType.SWAP_FILE_DELETED && swapType != RepositoryRecordType.SWAP_FILE_RENAMED) { + throw new IllegalArgumentException("swapType must be one of SWAP_IN, SWAP_OUT, SWAP_FILE_DELETED, or SWAP_FILE_RENAMED but was " + swapType); } + this.swapLocation = swapLocation; + setType(swapType); + } + + @Override + public String getOriginalSwapLocation() { + return originalSwapLocation; + } + + public void setSwapFileRenamed(final String originalSwapLocation, final String newSwapLocation) { + this.originalSwapLocation = originalSwapLocation; + this.swapLocation = newSwapLocation; + setType(RepositoryRecordType.SWAP_FILE_RENAMED); } @Override @@ -238,7 +252,7 @@ private void setType(final RepositoryRecordType newType) { return; } - if (this.type == RepositoryRecordType.CREATE) { + if (this.type == RepositoryRecordType.CREATE && getCurrent() != null) { // Because we don't copy updated attributes to `this.updatedAttributes` for CREATE records, we need to ensure // that if a record is changed from CREATE to anything else that we do properly update the `this.updatedAttributes` field. this.updatedAttributes = new HashMap<>(getCurrent().getAttributes()); diff --git a/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml b/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml index e5c7e30549b9..7468d198f903 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml +++ b/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml @@ -121,6 +121,8 @@ org.apache.nifi.asset.StandardAssetManager ./assets + org.apache.nifi.asset.StandardConnectorAssetManager + ./connector_assets diff --git a/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties b/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties index 6e536a67825a..afea4e093b25 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties +++ b/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties @@ -136,6 +136,11 @@ nifi.nar.persistence.provider.properties.directory=${nifi.nar.persistence.provid # Asset Management nifi.asset.manager.implementation=${nifi.asset.manager.implementation} nifi.asset.manager.properties.directory=${nifi.asset.manager.properties.directory} +nifi.connector.asset.manager.implementation=${nifi.connector.asset.manager.implementation} +nifi.connector.asset.manager.properties.directory=${nifi.connector.asset.manager.properties.directory} + +# Connector Configuration Provider +nifi.components.connectors.configuration.provider.implementation= # Site to Site properties nifi.remote.input.host= diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java index dc0ddf33048c..706a349d0861 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java @@ -61,6 +61,7 @@ import org.apache.nifi.util.FileUtils; import org.apache.nifi.util.NiFiProperties; import org.apache.nifi.web.ContentAccess; +import org.apache.nifi.web.NiFiConnectorWebContext; import org.apache.nifi.web.NiFiWebConfigurationContext; import org.apache.nifi.web.UiExtensionType; import org.apache.nifi.web.server.filter.FilterParameter; @@ -96,6 +97,7 @@ import java.io.BufferedWriter; import java.io.File; import java.io.IOException; +import java.io.InputStream; import java.io.InputStreamReader; import java.io.OutputStreamWriter; import java.lang.reflect.InvocationTargetException; @@ -114,6 +116,7 @@ import java.util.List; import java.util.Map; import java.util.Objects; +import java.util.Properties; import java.util.Set; import java.util.UUID; import java.util.concurrent.BlockingQueue; @@ -194,6 +197,7 @@ public class JettyServer implements NiFiServer, ExtensionUiLoader { // component (processor, controller service, reporting task) ui extensions private UiExtensionMapping componentUiExtensions; private Collection componentUiExtensionWebContexts; + private Collection connectorUiExtensionWebContexts; private final Map> webAppsByBundleCoordinate = new ConcurrentHashMap<>(); @@ -269,6 +273,7 @@ private Handler loadInitialWars(final Set bundles) { final ExtensionUiInfo extensionUiInfo = loadWars(otherWars); componentUiExtensionWebContexts = new ArrayList<>(extensionUiInfo.componentUiExtensionWebContexts()); contentViewerWebContexts = new ArrayList<>(extensionUiInfo.contentViewerWebContexts()); + connectorUiExtensionWebContexts = new ArrayList<>(extensionUiInfo.connectorUiExtensionWebContexts()); contentViewers = new HashSet<>(extensionUiInfo.contentViewers()); componentUiExtensions = new UiExtensionMapping(extensionUiInfo.componentUiExtensionsByType()); @@ -328,15 +333,18 @@ private void processExtensionUiBundle(final Bundle bundle) { final Collection componentUiExtensionWebContexts = extensionUiInfo.componentUiExtensionWebContexts(); final Collection contentViewerWebContexts = extensionUiInfo.contentViewerWebContexts(); + final Collection connectorUiExtensionWebContexts = extensionUiInfo.connectorUiExtensionWebContexts(); // Inject the configuration context and security filter into contexts that need it final ServletContext webApiServletContext = webApiContext.getServletHandler().getServletContext(); final WebApplicationContext webApplicationContext = WebApplicationContextUtils.getRequiredWebApplicationContext(webApiServletContext); final NiFiWebConfigurationContext configurationContext = webApplicationContext.getBean("nifiWebConfigurationContext", NiFiWebConfigurationContext.class); + final NiFiConnectorWebContext connectorWebContext = webApplicationContext.getBean("nifiConnectorWebContext", NiFiConnectorWebContext.class); final FilterHolder securityFilter = webApiContext.getServletHandler().getFilter(SPRING_SECURITY_FILTER_CHAIN); performInjectionForComponentUis(componentUiExtensionWebContexts, configurationContext, securityFilter); performInjectionForContentViewerUis(contentViewerWebContexts, webApplicationContext, securityFilter); + performInjectionForConnectorUis(connectorUiExtensionWebContexts, connectorWebContext, securityFilter); // Merge results of current loading into previously loaded results... this.componentUiExtensionWebContexts.addAll(componentUiExtensionWebContexts); @@ -417,6 +425,7 @@ private ExtensionUiInfo loadWars(final Map warToBundleLookup) { final Map mimeMappings = new HashMap<>(); final Collection componentUiExtensionWebContexts = new ArrayList<>(); final Collection contentViewerWebContexts = new ArrayList<>(); + final Collection connectorUiExtensionWebContexts = new ArrayList<>(); final Map> componentUiExtensionsByType = new HashMap<>(); final Map contentViewerServletContexts = new HashMap<>(); final Map> webAppContextsByBundleCoordinate = new HashMap<>(); @@ -433,8 +442,11 @@ private ExtensionUiInfo loadWars(final Map warToBundleLookup) { final Map> uiExtensionInWar = new HashMap<>(); identifyUiExtensionsForComponents(uiExtensionInWar, war); + // identify connector UI extensions from manifest (has route information) + final Map> connectorExtensions = readConnectorUiExtensions(war); + // only include wars that are for custom processor ui's - if (!uiExtensionInWar.isEmpty()) { + if (!uiExtensionInWar.isEmpty() || !connectorExtensions.isEmpty()) { // get the context path String warName = StringUtils.substringBeforeLast(war.getName(), "."); String warContextPath = String.format("/%s", warName); @@ -501,6 +513,35 @@ private ExtensionUiInfo loadWars(final Map warToBundleLookup) { } } + // process connector UI extensions with route information + for (final Map.Entry> connectorEntry : connectorExtensions.entrySet()) { + final String connectorType = connectorEntry.getKey(); + final Map routes = connectorEntry.getValue(); + + // build a bundle-specific key to support multiple versions of the same connector type + final BundleCoordinate coordinate = warBundle.getBundleDetails().getCoordinate(); + final String bundleSpecificKey = connectorType + ":" + coordinate.getGroup() + ":" + coordinate.getId() + ":" + coordinate.getVersion(); + + logger.info("Loading Connector UI extension [{}, {}] for {} with routes {}", UiExtensionType.Connector, warContextPath, bundleSpecificKey, routes.keySet()); + + // record the extension definition with route information + final UiExtension uiExtension = new UiExtension(UiExtensionType.Connector, warContextPath, routes); + + // create if this is the first extension for this component type (using bundle-specific key) + final List componentUiExtensionsForType = componentUiExtensionsByType.computeIfAbsent(bundleSpecificKey, k -> new ArrayList<>()); + + // see if there is already a ui extension of this same type + if (containsUiExtensionType(componentUiExtensionsForType, UiExtensionType.Connector)) { + throw new IllegalStateException(String.format("Encountered duplicate Connector UI for %s", bundleSpecificKey)); + } + + // record this extension + componentUiExtensionsForType.add(uiExtension); + + // this ui extension provides a connector custom ui + connectorUiExtensionWebContexts.add(extensionUiContext); + } + // include custom ui web context in the handlers webAppContexts.add(extensionUiContext); @@ -512,7 +553,7 @@ private ExtensionUiInfo loadWars(final Map warToBundleLookup) { } return new ExtensionUiInfo(webAppContexts, mimeMappings, contentViewers, componentUiExtensionWebContexts, contentViewerWebContexts, - componentUiExtensionsByType, contentViewerServletContexts, webAppContextsByBundleCoordinate); + connectorUiExtensionWebContexts, componentUiExtensionsByType, contentViewerServletContexts, webAppContextsByBundleCoordinate); } /** @@ -593,6 +634,58 @@ private void identifyUiExtensionsForComponents(final Map + * # connector-type.route-name=route-path + * org.apache.nifi.connectors.KafkaToS3.configuration=#/configure + * org.apache.nifi.connectors.KafkaToS3.details=#/details + * + * + * @param warFile the WAR file to inspect + * @return a map of connector type to route map (route name to route path), or empty map if no manifest found + */ + private Map> readConnectorUiExtensions(final File warFile) { + final Map> connectorExtensions = new HashMap<>(); + + try (final JarFile jarFile = new JarFile(warFile)) { + final JarEntry propertiesEntry = jarFile.getJarEntry("META-INF/nifi-connector"); + if (propertiesEntry == null) { + return connectorExtensions; + } + + try (final InputStream inputStream = jarFile.getInputStream(propertiesEntry)) { + final Properties properties = new Properties(); + properties.load(inputStream); + + for (final String propertyName : properties.stringPropertyNames()) { + final String propertyValue = properties.getProperty(propertyName); + if (propertyValue == null || propertyValue.isBlank()) { + continue; + } + + // Parse property name: connector-type.route-name + final int lastDotIndex = propertyName.lastIndexOf('.'); + if (lastDotIndex <= 0 || lastDotIndex >= propertyName.length() - 1) { + logger.warn("Invalid connector UI property format: {}. Expected format: connector-type.route-name=route-path", propertyName); + continue; + } + + final String connectorType = propertyName.substring(0, lastDotIndex); + final String routeName = propertyName.substring(lastDotIndex + 1); + + final Map routes = connectorExtensions.computeIfAbsent(connectorType, k -> new HashMap<>()); + routes.put(routeName, propertyValue); + } + } + } catch (final IOException ioe) { + logger.warn("Unable to inspect {} for Connector UI extensions.", warFile, ioe); + } + + return connectorExtensions; + } + /** * Extracts the component type. Trims the line and considers comments. * Returns null if no type was found. @@ -803,11 +896,15 @@ public void start() { // get the application context final WebApplicationContext webApplicationContext = WebApplicationContextUtils.getRequiredWebApplicationContext(webApiServletContext); final NiFiWebConfigurationContext configurationContext = webApplicationContext.getBean("nifiWebConfigurationContext", NiFiWebConfigurationContext.class); + final NiFiConnectorWebContext connectorWebContext = webApplicationContext.getBean("nifiConnectorWebContext", NiFiConnectorWebContext.class); final FilterHolder securityFilter = webApiContext.getServletHandler().getFilter(SPRING_SECURITY_FILTER_CHAIN); // component ui extensions performInjectionForComponentUis(componentUiExtensionWebContexts, configurationContext, securityFilter); + // connector ui extensions + performInjectionForConnectorUis(connectorUiExtensionWebContexts, connectorWebContext, securityFilter); + // content viewer extensions performInjectionForContentViewerUis(contentViewerWebContexts, webApplicationContext, securityFilter); webApiServletContext.setAttribute("content-viewers", contentViewers); @@ -948,6 +1045,21 @@ private void performInjectionForComponentUis(final Collection com } } + private void performInjectionForConnectorUis(final Collection connectorUiExtensionWebContexts, + final NiFiConnectorWebContext connectorWebContext, + final FilterHolder securityFilter) { + for (final WebAppContext customUiContext : connectorUiExtensionWebContexts) { + // set the connector web context in each connector custom ui servlet context + final ServletContext customUiServletContext = customUiContext.getServletHandler().getServletContext(); + customUiServletContext.setAttribute("nifi-connector-web-context", connectorWebContext); + + // add the security filter to any ui extensions wars + if (securityFilter != null) { + customUiContext.addFilter(securityFilter, "/*", EnumSet.allOf(DispatcherType.class)); + } + } + } + private void performInjectionForContentViewerUis(final Collection contentViewerWebContexts, final WebApplicationContext webApiApplicationContext, final FilterHolder securityFilter) { @@ -1017,6 +1129,7 @@ private ErrorPageErrorHandler getErrorHandler() { private record ExtensionUiInfo(Collection webAppContexts, Map mimeMappings, Collection contentViewers, Collection componentUiExtensionWebContexts, Collection contentViewerWebContexts, + Collection connectorUiExtensionWebContexts, Map> componentUiExtensionsByType, Map contentViewerServletContexts, Map> webAppContextsByBundleCoordinate) { diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/test/java/org/apache/nifi/web/server/StandardServerProviderTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/test/java/org/apache/nifi/web/server/StandardServerProviderTest.java index 6d7b38847cad..8b18bfeb5070 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/test/java/org/apache/nifi/web/server/StandardServerProviderTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/test/java/org/apache/nifi/web/server/StandardServerProviderTest.java @@ -123,7 +123,7 @@ static void setConfiguration() throws Exception { void testGetServer() { final Properties applicationProperties = new Properties(); applicationProperties.setProperty(NiFiProperties.WEB_HTTP_PORT, RANDOM_PORT); - final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties(null, applicationProperties); + final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties((String) null, applicationProperties); final StandardServerProvider provider = new StandardServerProvider(null); @@ -137,7 +137,7 @@ void testGetServer() { void testGetServerHttps() { final Properties applicationProperties = new Properties(); applicationProperties.setProperty(NiFiProperties.WEB_HTTPS_PORT, RANDOM_PORT); - final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties(null, applicationProperties); + final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties((String) null, applicationProperties); final StandardServerProvider provider = new StandardServerProvider(sslContext); @@ -151,7 +151,7 @@ void testGetServerHttps() { void testGetServerStart() throws Exception { final Properties applicationProperties = new Properties(); applicationProperties.setProperty(NiFiProperties.WEB_HTTP_PORT, RANDOM_PORT); - final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties(null, applicationProperties); + final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties((String) null, applicationProperties); final StandardServerProvider provider = new StandardServerProvider(null); @@ -175,7 +175,7 @@ void testGetServerHttpsRequestsCompleted() throws Exception { final Properties applicationProperties = new Properties(); applicationProperties.setProperty(NiFiProperties.WEB_HTTPS_PORT, RANDOM_PORT); applicationProperties.setProperty(NiFiProperties.WEB_PROXY_HOST, PUBLIC_HOST); - final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties(null, applicationProperties); + final NiFiProperties properties = NiFiProperties.createBasicNiFiProperties((String) null, applicationProperties); final StandardServerProvider provider = new StandardServerProvider(sslContext); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java index da921b4535dc..11268a99c4fc 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-ui-extension/src/main/java/org/apache/nifi/ui/extension/UiExtension.java @@ -18,6 +18,8 @@ import org.apache.nifi.web.UiExtensionType; +import java.util.Map; + /** * Information about a UI extension required to be invoked. */ @@ -25,10 +27,16 @@ public class UiExtension { private final UiExtensionType extensionType; private final String contextPath; + private final Map supportedRoutes; public UiExtension(final UiExtensionType extensionType, final String contextPath) { + this(extensionType, contextPath, null); + } + + public UiExtension(final UiExtensionType extensionType, final String contextPath, final Map supportedRoutes) { this.extensionType = extensionType; this.contextPath = contextPath; + this.supportedRoutes = supportedRoutes; } /** @@ -45,4 +53,12 @@ public String getContextPath() { return contextPath; } + /** + * @return A map of route names to route paths for this UI extension, or null if routes are not applicable. + * Route paths may include a hash prefix for hash-based routing (e.g., "#/wizard") or be path-based (e.g., "/wizard"). + */ + public Map getSupportedRoutes() { + return supportedRoutes; + } + } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/audit/ConnectorAuditor.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/audit/ConnectorAuditor.java new file mode 100644 index 000000000000..57c762451fe3 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/audit/ConnectorAuditor.java @@ -0,0 +1,416 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.audit; + +import org.apache.nifi.action.Action; +import org.apache.nifi.action.Component; +import org.apache.nifi.action.FlowChangeAction; +import org.apache.nifi.action.Operation; +import org.apache.nifi.action.component.details.FlowChangeExtensionDetails; +import org.apache.nifi.action.details.ActionDetails; +import org.apache.nifi.action.details.FlowChangeConfigureDetails; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConnectorConfiguration; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.NamedStepConfiguration; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.components.connector.StringLiteralValue; +import org.apache.nifi.web.api.dto.AssetReferenceDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.dao.ConnectorDAO; +import org.aspectj.lang.ProceedingJoinPoint; +import org.aspectj.lang.annotation.Around; +import org.aspectj.lang.annotation.Aspect; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +import java.util.ArrayList; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +/** + * Audits connector creation/removal and configuration changes. + */ +@Service +@Aspect +public class ConnectorAuditor extends NiFiAuditor { + + private static final Logger logger = LoggerFactory.getLogger(ConnectorAuditor.class); + + /** + * Audits the creation of connectors via createConnector(). + * + * @param proceedingJoinPoint join point + * @return connector node + * @throws Throwable if an error occurs + */ + @Around("within(org.apache.nifi.web.dao.ConnectorDAO+) && " + + "execution(org.apache.nifi.components.connector.ConnectorNode createConnector(..))") + public ConnectorNode createConnectorAdvice(final ProceedingJoinPoint proceedingJoinPoint) throws Throwable { + final ConnectorNode connector = (ConnectorNode) proceedingJoinPoint.proceed(); + + final Action action = generateAuditRecord(connector, Operation.Add); + if (action != null) { + saveAction(action, logger); + } + + return connector; + } + + /** + * Audits the removal of a connector via deleteConnector(). + * + * @param proceedingJoinPoint join point + * @param connectorId connector id + * @param connectorDAO connector dao + * @throws Throwable if an error occurs + */ + @Around("within(org.apache.nifi.web.dao.ConnectorDAO+) && " + + "execution(void deleteConnector(java.lang.String)) && " + + "args(connectorId) && " + + "target(connectorDAO)") + public void removeConnectorAdvice(final ProceedingJoinPoint proceedingJoinPoint, final String connectorId, final ConnectorDAO connectorDAO) throws Throwable { + final ConnectorNode connector = connectorDAO.getConnector(connectorId); + + proceedingJoinPoint.proceed(); + + final Action action = generateAuditRecord(connector, Operation.Remove); + if (action != null) { + saveAction(action, logger); + } + } + + /** + * Audits the starting of a connector via startConnector(). + * + * @param proceedingJoinPoint join point + * @param connectorId connector id + * @param connectorDAO connector dao + * @throws Throwable if an error occurs + */ + @Around("within(org.apache.nifi.web.dao.ConnectorDAO+) && " + + "execution(void startConnector(java.lang.String)) && " + + "args(connectorId) && " + + "target(connectorDAO)") + public void startConnectorAdvice(final ProceedingJoinPoint proceedingJoinPoint, final String connectorId, final ConnectorDAO connectorDAO) throws Throwable { + final ConnectorNode connector = connectorDAO.getConnector(connectorId); + final ConnectorState previousState = connector.getCurrentState(); + + proceedingJoinPoint.proceed(); + + if (isAuditable() && previousState != ConnectorState.RUNNING && previousState != ConnectorState.STARTING) { + final Action action = generateAuditRecord(connector, Operation.Start); + if (action != null) { + saveAction(action, logger); + } + } + } + + /** + * Audits the stopping of a connector via stopConnector(). + * + * @param proceedingJoinPoint join point + * @param connectorId connector id + * @param connectorDAO connector dao + * @throws Throwable if an error occurs + */ + @Around("within(org.apache.nifi.web.dao.ConnectorDAO+) && " + + "execution(void stopConnector(java.lang.String)) && " + + "args(connectorId) && " + + "target(connectorDAO)") + public void stopConnectorAdvice(final ProceedingJoinPoint proceedingJoinPoint, final String connectorId, final ConnectorDAO connectorDAO) throws Throwable { + final ConnectorNode connector = connectorDAO.getConnector(connectorId); + final ConnectorState previousState = connector.getCurrentState(); + + proceedingJoinPoint.proceed(); + + if (isAuditable() && previousState != ConnectorState.STOPPED && previousState != ConnectorState.STOPPING) { + final Action action = generateAuditRecord(connector, Operation.Stop); + if (action != null) { + saveAction(action, logger); + } + } + } + + /** + * Audits configuration step updates via updateConnectorConfigurationStep(). + * + * @param proceedingJoinPoint join point + * @param connectorId connector id + * @param configurationStepName name of the configuration step + * @param configurationStepConfiguration the configuration step configuration DTO + * @param connectorDAO connector dao + * @throws Throwable if an error occurs + */ + @Around("within(org.apache.nifi.web.dao.ConnectorDAO+) && " + + "execution(void updateConnectorConfigurationStep(java.lang.String, java.lang.String, org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO)) && " + + "args(connectorId, configurationStepName, configurationStepConfiguration) && " + + "target(connectorDAO)") + public void updateConfigurationStepAdvice(final ProceedingJoinPoint proceedingJoinPoint, final String connectorId, final String configurationStepName, + final ConfigurationStepConfigurationDTO configurationStepConfiguration, final ConnectorDAO connectorDAO) throws Throwable { + final ConnectorNode connector = connectorDAO.getConnector(connectorId); + + // Capture the current property values before the update (flat map: property name -> value) + final Map previousValues = extractCurrentPropertyValues(connector, configurationStepName); + + proceedingJoinPoint.proceed(); + + if (isAuditable()) { + // Generate audit actions for each changed property, using the DTO for group structure + final List actions = generateConfigurationChangeActions(connector, configurationStepName, previousValues, configurationStepConfiguration); + if (!actions.isEmpty()) { + saveActions(actions, logger); + } + } + } + + /** + * Extracts the current property values for a specific configuration step from the connector's working flow context. + * + * @param connector the connector node + * @param configurationStepName the name of the configuration step + * @return a map of property name to property value + */ + private Map extractCurrentPropertyValues(final ConnectorNode connector, final String configurationStepName) { + final Map result = new HashMap<>(); + + final ConnectorConfiguration configuration = connector.getWorkingFlowContext().getConfigurationContext().toConnectorConfiguration(); + for (final NamedStepConfiguration namedStepConfig : configuration.getNamedStepConfigurations()) { + if (Objects.equals(namedStepConfig.stepName(), configurationStepName)) { + final StepConfiguration stepConfig = namedStepConfig.configuration(); + for (final Map.Entry entry : stepConfig.getPropertyValues().entrySet()) { + result.put(entry.getKey(), formatValueReference(entry.getValue())); + } + break; + } + } + + return result; + } + + /** + * Formats a ConnectorValueReference as a string for audit logging. + * + * @param valueRef the value reference + * @return the formatted string representation + */ + private String formatValueReference(final ConnectorValueReference valueRef) { + if (valueRef == null) { + return null; + } + + return switch (valueRef) { + case StringLiteralValue stringLiteral -> stringLiteral.getValue(); + case AssetReference assetRef -> "assets:" + assetRef.getAssetIdentifiers(); + case SecretReference secretRef -> "secret:" + secretRef.getProviderId() + "/" + secretRef.getProviderName() + "/" + secretRef.getSecretName(); + }; + } + + /** + * Formats a ConnectorValueReferenceDTO as a string for audit logging. + * + * @param valueRefDto the value reference DTO + * @return the formatted string representation + */ + private String formatValueReferenceDto(final ConnectorValueReferenceDTO valueRefDto) { + if (valueRefDto == null) { + return null; + } + + final String valueType = valueRefDto.getValueType(); + if (valueType == null || "STRING_LITERAL".equals(valueType)) { + return valueRefDto.getValue(); + } else if ("ASSET_REFERENCE".equals(valueType)) { + return "assets:" + Stream.ofNullable(valueRefDto.getAssetReferences()) + .flatMap(List::stream) + .map(AssetReferenceDTO::getId) + .collect(Collectors.joining(",")); + } else if ("SECRET_REFERENCE".equals(valueType)) { + return "secret:" + valueRefDto.getSecretProviderId() + "/" + valueRefDto.getSecretProviderName() + "/" + valueRefDto.getSecretName(); + } + + return valueRefDto.getValue(); + } + + /** + * Generates audit actions for configuration changes by comparing previous values with the new values from the DTO. + * + * @param connector the connector node + * @param configurationStepName the configuration step name + * @param previousValues the previous property values (flat map: property name -> value) + * @param configurationStepDto the configuration step DTO containing new values with group structure + * @return list of actions for changed properties + */ + private List generateConfigurationChangeActions(final ConnectorNode connector, final String configurationStepName, + final Map previousValues, final ConfigurationStepConfigurationDTO configurationStepDto) { + final List actions = new ArrayList<>(); + final Date timestamp = new Date(); + + final FlowChangeExtensionDetails connectorDetails = new FlowChangeExtensionDetails(); + connectorDetails.setType(connector.getComponentType()); + + if (configurationStepDto.getPropertyGroupConfigurations() == null) { + return actions; + } + + // Iterate through all property groups in the DTO to preserve group names for audit logging + for (final PropertyGroupConfigurationDTO groupDto : configurationStepDto.getPropertyGroupConfigurations()) { + final String groupName = groupDto.getPropertyGroupName(); + + if (groupDto.getPropertyValues() == null) { + continue; + } + + // Check each property in this group + for (final Map.Entry propertyEntry : groupDto.getPropertyValues().entrySet()) { + final String propertyName = propertyEntry.getKey(); + final String newValue = formatValueReferenceDto(propertyEntry.getValue()); + final String previousValue = previousValues.get(propertyName); + + // Only create an action if the value changed + if (!Objects.equals(previousValue, newValue)) { + final String fullPropertyName = formatPropertyName(configurationStepName, groupName, propertyName); + + final FlowChangeConfigureDetails actionDetails = new FlowChangeConfigureDetails(); + actionDetails.setName(fullPropertyName); + actionDetails.setPreviousValue(previousValue); + actionDetails.setValue(newValue); + + final FlowChangeAction configurationAction = createFlowChangeAction(); + configurationAction.setOperation(Operation.Configure); + configurationAction.setTimestamp(timestamp); + configurationAction.setSourceId(connector.getIdentifier()); + configurationAction.setSourceName(connector.getName()); + configurationAction.setSourceType(Component.Connector); + configurationAction.setComponentDetails(connectorDetails); + configurationAction.setActionDetails(actionDetails); + + actions.add(configurationAction); + } + } + } + + return actions; + } + + /** + * Formats the property name for audit logging, including step and group context. + * + * @param stepName the configuration step name + * @param groupName the property group name + * @param propertyName the property name + * @return the formatted property name + */ + private String formatPropertyName(final String stepName, final String groupName, final String propertyName) { + if (groupName == null || groupName.isEmpty()) { + return stepName + " / " + propertyName; + } + return stepName + " / " + groupName + " / " + propertyName; + } + + /** + * Audits application of connector updates via applyConnectorUpdate(). + * + * @param proceedingJoinPoint join point + * @param connectorId connector id + * @param connectorDAO connector dao + * @throws Throwable if an error occurs + */ + @Around("within(org.apache.nifi.web.dao.ConnectorDAO+) && " + + "execution(void applyConnectorUpdate(java.lang.String)) && " + + "args(connectorId) && " + + "target(connectorDAO)") + public void applyConnectorUpdateAdvice(final ProceedingJoinPoint proceedingJoinPoint, final String connectorId, final ConnectorDAO connectorDAO) throws Throwable { + final ConnectorNode connector = connectorDAO.getConnector(connectorId); + + proceedingJoinPoint.proceed(); + + if (isAuditable()) { + final FlowChangeExtensionDetails connectorDetails = new FlowChangeExtensionDetails(); + connectorDetails.setType(connector.getComponentType()); + + final FlowChangeConfigureDetails actionDetails = new FlowChangeConfigureDetails(); + actionDetails.setName("Applied Update"); + actionDetails.setValue("true"); + actionDetails.setPreviousValue(null); + + final FlowChangeAction configurationAction = createFlowChangeAction(); + configurationAction.setOperation(Operation.Configure); + configurationAction.setTimestamp(new Date()); + configurationAction.setSourceId(connector.getIdentifier()); + configurationAction.setSourceName(connector.getName()); + configurationAction.setSourceType(Component.Connector); + configurationAction.setComponentDetails(connectorDetails); + configurationAction.setActionDetails(actionDetails); + + saveAction(configurationAction, logger); + } + } + + /** + * Generates an audit record for a connector. + * + * @param connector the connector + * @param operation the operation + * @return the action + */ + public Action generateAuditRecord(final ConnectorNode connector, final Operation operation) { + return generateAuditRecord(connector, operation, null); + } + + /** + * Generates an audit record for a connector. + * + * @param connector the connector + * @param operation the operation + * @param actionDetails the action details + * @return the action + */ + public Action generateAuditRecord(final ConnectorNode connector, final Operation operation, final ActionDetails actionDetails) { + FlowChangeAction action = null; + + if (isAuditable()) { + final FlowChangeExtensionDetails connectorDetails = new FlowChangeExtensionDetails(); + connectorDetails.setType(connector.getComponentType()); + + action = createFlowChangeAction(); + action.setOperation(operation); + action.setSourceId(connector.getIdentifier()); + action.setSourceName(connector.getName()); + action.setSourceType(Component.Connector); + action.setComponentDetails(connectorDetails); + + if (actionDetails != null) { + action.setActionDetails(actionDetails); + } + } + + return action; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/AuthorizableLookup.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/AuthorizableLookup.java index de725ad0cacf..382126b6754e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/AuthorizableLookup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/AuthorizableLookup.java @@ -137,6 +137,15 @@ public interface AuthorizableLookup { */ ConnectionAuthorizable getConnection(String id); + /** + * Get the authorizable Connection, optionally including Connector-managed ProcessGroups in the search. + * + * @param id connection id + * @param includeConnectorManaged whether to search Connector-managed ProcessGroups + * @return authorizable + */ + ConnectionAuthorizable getConnection(String id, boolean includeConnectorManaged); + /** * Get the authorizable root ProcessGroup. * @@ -152,6 +161,15 @@ public interface AuthorizableLookup { */ ProcessGroupAuthorizable getProcessGroup(String id); + /** + * Get the authorizable ProcessGroup, optionally including Connector-managed ProcessGroups in the search. + * + * @param id process group id + * @param includeConnectorManaged whether to search Connector-managed ProcessGroups + * @return authorizable + */ + ProcessGroupAuthorizable getProcessGroup(String id, boolean includeConnectorManaged); + /** * Get the authorizable RemoteProcessGroup. * @@ -256,6 +274,12 @@ public interface AuthorizableLookup { */ Authorizable getParameterContexts(); + /** + * Get the authorizable for Connectors + * @return authorizable + */ + Authorizable getConnectors(); + /** * Get the authorizable connectable. Note this does not include RemoteGroupPorts. * @@ -309,6 +333,13 @@ public interface AuthorizableLookup { */ Authorizable getAuthorizableFromResource(final String resource); + /** + * Get the authorizable for the connector with the given ID + * @param connectorId the ID of the connector + * @return authorizable + */ + Authorizable getConnector(String connectorId); + /** * Get the authorizable for access to the System resource. * diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/StandardAuthorizableLookup.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/StandardAuthorizableLookup.java index 15d150757216..6e59e3442ecc 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/StandardAuthorizableLookup.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/StandardAuthorizableLookup.java @@ -58,6 +58,7 @@ import org.apache.nifi.web.controller.ControllerFacade; import org.apache.nifi.web.dao.AccessPolicyDAO; import org.apache.nifi.web.dao.ConnectionDAO; +import org.apache.nifi.web.dao.ConnectorDAO; import org.apache.nifi.web.dao.ControllerServiceDAO; import org.apache.nifi.web.dao.FlowAnalysisRuleDAO; import org.apache.nifi.web.dao.FlowRegistryDAO; @@ -123,6 +124,7 @@ public Resource getResource() { } }; + private static final Authorizable RESOURCE_AUTHORIZABLE = new Authorizable() { @Override public Authorizable getParentAuthorizable() { @@ -184,6 +186,7 @@ public Resource getResource() { private PortDAO inputPortDAO; private PortDAO outputPortDAO; private ConnectionDAO connectionDAO; + private ConnectorDAO connectorDAO; private ControllerServiceDAO controllerServiceDAO; private ReportingTaskDAO reportingTaskDAO; private FlowAnalysisRuleDAO flowAnalysisRuleDAO; @@ -291,7 +294,12 @@ public ParameterContext getParameterContext(final String id) { @Override public ConnectionAuthorizable getConnection(final String id) { - final Connection connection = connectionDAO.getConnection(id); + return getConnection(id, false); + } + + @Override + public ConnectionAuthorizable getConnection(final String id, final boolean includeConnectorManaged) { + final Connection connection = connectionDAO.getConnection(id, includeConnectorManaged); return new StandardConnectionAuthorizable(connection); } @@ -302,7 +310,12 @@ public ProcessGroupAuthorizable getRootProcessGroup() { @Override public ProcessGroupAuthorizable getProcessGroup(final String id) { - final ProcessGroup processGroup = processGroupDAO.getProcessGroup(id); + return getProcessGroup(id, false); + } + + @Override + public ProcessGroupAuthorizable getProcessGroup(final String id, final boolean includeConnectorManaged) { + final ProcessGroup processGroup = processGroupDAO.getProcessGroup(id, includeConnectorManaged); return new StandardProcessGroupAuthorizable(processGroup, controllerFacade.getExtensionManager()); } @@ -605,6 +618,11 @@ public Authorizable getAuthorizableFromResource(final String resource) { } } + @Override + public Authorizable getConnector(final String connectorId) { + return connectorDAO.getConnector(connectorId); + } + private Authorizable handleResourceTypeContainingOtherResourceType(final String resource, final ResourceType resourceType) { // get the resource type final String baseResource = StringUtils.substringAfter(resource, resourceType.getValue()); @@ -648,6 +666,7 @@ private Authorizable getAccessPolicyByResource(final ResourceType resourceType, case FlowAnalysisRule -> getFlowAnalysisRule(componentId).getAuthorizable(); case ParameterContext -> getParameterContext(componentId); case ParameterProvider -> getParameterProvider(componentId).getAuthorizable(); + case Connector -> getConnector(componentId); default -> null; }; @@ -734,6 +753,9 @@ public Resource getResource() { case ParameterContext: authorizable = getParameterContexts(); break; + case Connector: + authorizable = getConnectors(); + break; } if (authorizable == null) { @@ -826,6 +848,21 @@ public Resource getResource() { }; } + @Override + public Authorizable getConnectors() { + return new Authorizable() { + @Override + public Authorizable getParentAuthorizable() { + return null; + } + + @Override + public Resource getResource() { + return ResourceFactory.getConnectorsResource(); + } + }; + } + /** * ComponentAuthorizable for a ConfigurableComponent. This authorizable is intended only to be used when * creating new components. @@ -1407,6 +1444,11 @@ public void setConnectionDAO(ConnectionDAO connectionDAO) { this.connectionDAO = connectionDAO; } + @Autowired + public void setConnectorDAO(ConnectorDAO connectorDAO) { + this.connectorDAO = connectorDAO; + } + @Autowired public void setControllerServiceDAO(ControllerServiceDAO controllerServiceDAO) { this.controllerServiceDAO = controllerServiceDAO; diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiServiceFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiServiceFacade.java index 3b5babdc0692..cd7d6897a4d1 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiServiceFacade.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiServiceFacade.java @@ -17,10 +17,12 @@ package org.apache.nifi.web; import io.prometheus.client.CollectorRegistry; +import org.apache.nifi.asset.Asset; import org.apache.nifi.authorization.AuthorizeAccess; import org.apache.nifi.authorization.RequestAction; import org.apache.nifi.authorization.user.NiFiUser; import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.FlowAnalysisRuleDefinition; import org.apache.nifi.c2.protocol.component.api.FlowRegistryClientDefinition; @@ -57,7 +59,9 @@ import org.apache.nifi.web.api.dto.ComponentHistoryDTO; import org.apache.nifi.web.api.dto.ComponentStateDTO; import org.apache.nifi.web.api.dto.ConfigVerificationResultDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; import org.apache.nifi.web.api.dto.ConnectionDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; import org.apache.nifi.web.api.dto.ControllerConfigurationDTO; import org.apache.nifi.web.api.dto.ControllerDTO; import org.apache.nifi.web.api.dto.ControllerServiceDTO; @@ -106,9 +110,13 @@ import org.apache.nifi.web.api.entity.ClearBulletinsResultEntity; import org.apache.nifi.web.api.entity.ComponentValidationResultEntity; import org.apache.nifi.web.api.entity.ConfigurationAnalysisEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; import org.apache.nifi.web.api.entity.ConnectionEntity; import org.apache.nifi.web.api.entity.ConnectionStatisticsEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; import org.apache.nifi.web.api.entity.ControllerBulletinsEntity; import org.apache.nifi.web.api.entity.ControllerConfigurationEntity; import org.apache.nifi.web.api.entity.ControllerServiceEntity; @@ -149,6 +157,7 @@ import org.apache.nifi.web.api.entity.RemoteProcessGroupStatusEntity; import org.apache.nifi.web.api.entity.ReportingTaskEntity; import org.apache.nifi.web.api.entity.ScheduleComponentsEntity; +import org.apache.nifi.web.api.entity.SecretsEntity; import org.apache.nifi.web.api.entity.SnippetEntity; import org.apache.nifi.web.api.entity.StartVersionControlRequestEntity; import org.apache.nifi.web.api.entity.StatusHistoryEntity; @@ -170,6 +179,7 @@ import java.util.Date; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.function.Function; import java.util.function.Supplier; @@ -178,6 +188,84 @@ * Defines the NiFiServiceFacade interface. */ public interface NiFiServiceFacade { + // ---------------------------------------- + // Connector methods + // ---------------------------------------- + + void verifyCreateConnector(ConnectorDTO connectorDTO); + + ConnectorEntity createConnector(Revision revision, ConnectorDTO connectorDTO); + + Set getConnectors(); + + ConnectorEntity getConnector(String id); + + void verifyUpdateConnector(ConnectorDTO connectorDTO); + + ConnectorEntity updateConnector(Revision revision, ConnectorDTO connectorDTO); + + void verifyDeleteConnector(String id); + + ConnectorEntity deleteConnector(Revision revision, String id); + + ConnectorEntity scheduleConnector(Revision revision, String id, ScheduledState state); + + void verifyDrainConnector(String id); + + ConnectorEntity drainConnector(Revision revision, String id); + + void verifyCancelConnectorDrain(String id); + + ConnectorEntity cancelConnectorDrain(Revision revision, String id); + + ConfigurationStepNamesEntity getConnectorConfigurationSteps(String id); + + ConfigurationStepEntity getConnectorConfigurationStep(String id, String configurationStepName); + + ConfigurationStepEntity updateConnectorConfigurationStep(Revision revision, String id, String configurationStepName, ConfigurationStepConfigurationDTO configurationStepConfiguration); + + ConnectorEntity applyConnectorUpdate(Revision revision, String connectorId); + + ConnectorEntity discardConnectorUpdate(Revision revision, String connectorId); + + ProcessGroupFlowEntity getConnectorFlow(String connectorId, String processGroupId, boolean uiOnly); + + ProcessGroupStatusEntity getConnectorProcessGroupStatus(String id, Boolean recursive); + + Set getConnectorControllerServices(String connectorId, String processGroupId, boolean includeAncestorGroups, + boolean includeDescendantGroups, boolean includeReferencingComponents); + + void verifyCanVerifyConnectorConfigurationStep(String connectorId, String configurationStepName); + + List performConnectorConfigurationStepVerification(String connectorId, String configurationStepName, ConfigurationStepConfigurationDTO configurationStepConfiguration); + + SearchResultsDTO searchConnector(String connectorId, String query); + + ConnectorPropertyAllowableValuesEntity getConnectorPropertyAllowableValues(String connectorId, String stepName, String groupName, String propertyName, String filter); + + void verifyCreateConnectorAsset(String connectorId); + + AssetEntity createConnectorAsset(String connectorId, String assetId, String assetName, InputStream content) throws IOException; + + List getConnectorAssets(String connectorId); + + Optional getConnectorAsset(String assetId); + + /** + * Verifies that the connector is in a state where FlowFiles can be purged. + * + * @param connectorId the connector ID + * @throws IllegalStateException if the connector is not in a state where FlowFiles can be purged + */ + void verifyPurgeConnectorFlowFiles(String connectorId); + + /** + * Purges all FlowFiles from the connector. + * + * @param connectorId the connector ID + * @param requestor the identity of the user requesting the purge (used for provenance events) + */ + void purgeConnectorFlowFiles(String connectorId, String requestor); // ---------------------------------------- // Synchronization methods @@ -475,6 +563,16 @@ public interface NiFiServiceFacade { */ Set getProcessorTypes(final String bundleGroupFilter, final String bundleArtifactFilter, final String typeFilter); + /** + * Returns the list of connector types. + * + * @param bundleGroupFilter if specified, must be member of bundle group + * @param bundleArtifactFilter if specified, must be member of bundle artifact + * @param typeFilter if specified, type must match + * @return The list of available connector types matching specified criteria + */ + Set getConnectorTypes(final String bundleGroupFilter, final String bundleArtifactFilter, final String typeFilter); + /** * Returns the list of controller service types. * @@ -571,6 +669,17 @@ Set getControllerServiceTypes(final String serviceType, final */ FlowAnalysisRuleDefinition getFlowAnalysisRuleDefinition(String group, String artifact, String version, String type); + /** + * Return the ConnectorDefinition for the specified Connector. + * + * @param group The bundle group + * @param artifact The bundle artifact + * @param version The bundle version + * @param type The Connector type + * @return The ConnectorDefinition + */ + ConnectorDefinition getConnectorDefinition(String group, String artifact, String version, String type); + /** * Return the additionalDetails for the specified component. * @@ -582,6 +691,18 @@ Set getControllerServiceTypes(final String serviceType, final */ String getAdditionalDetails(String group, String artifact, String version, String type); + /** + * Return the step documentation for the specified Connector configuration step. + * + * @param group The bundle group + * @param artifact The bundle artifact + * @param version The bundle version + * @param connectorType The fully qualified class name of the Connector + * @param stepName The name of the configuration step + * @return The step documentation markdown content + */ + String getStepDocumentation(String group, String artifact, String version, String connectorType, String stepName); + /** * Returns the list of parameter provider types. * @@ -1975,6 +2096,60 @@ ProcessGroupEntity updateProcessGroupContents(Revision revision, String groupId, */ ComponentStateDTO getRemoteProcessGroupState(String remoteProcessGroupId); + /** + * Gets the state for a processor within a connector's managed process group. + * + * @param connectorId the connector id + * @param processorId the processor id + * @return the component state + */ + ComponentStateDTO getConnectorProcessorState(String connectorId, String processorId); + + /** + * Verifies the processor state within a connector could be cleared. + * + * @param connectorId the connector id + * @param processorId the processor id + */ + void verifyCanClearConnectorProcessorState(String connectorId, String processorId); + + /** + * Clears the state for a processor within a connector's managed process group. + * + * @param connectorId the connector id + * @param processorId the processor id + * @param componentStateDTO the state of the processor + * @return the cleared component state + */ + ComponentStateDTO clearConnectorProcessorState(String connectorId, String processorId, ComponentStateDTO componentStateDTO); + + /** + * Gets the state for a controller service within a connector's managed process group. + * + * @param connectorId the connector id + * @param controllerServiceId the controller service id + * @return the component state + */ + ComponentStateDTO getConnectorControllerServiceState(String connectorId, String controllerServiceId); + + /** + * Verifies the controller service state within a connector could be cleared. + * + * @param connectorId the connector id + * @param controllerServiceId the controller service id + */ + void verifyCanClearConnectorControllerServiceState(String connectorId, String controllerServiceId); + + /** + * Clears the state for a controller service within a connector's managed process group. + * + * @param connectorId the connector id + * @param controllerServiceId the controller service id + * @param componentStateDTO the state of the controller service + * @return the cleared component state + */ + ComponentStateDTO clearConnectorControllerServiceState(String connectorId, String controllerServiceId, ComponentStateDTO componentStateDTO); + // ---------------------------------------- // Label methods // ---------------------------------------- @@ -3132,4 +3307,16 @@ ControllerServiceReferencingComponentsEntity updateControllerServiceReferencingC * @return the list of listen Ports accessible to the current user */ Set getListenPorts(NiFiUser user); + + // ---------------------------------------- + // Secrets methods + // ---------------------------------------- + + /** + * Gets all secrets available from all secret providers. Note: The actual secret values are not included + * in the response for security reasons; only metadata is returned. + * + * @return the secrets entity containing metadata for all available secrets + */ + SecretsEntity getSecrets(); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiWebApiResourceConfig.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiWebApiResourceConfig.java index 3baf1b987522..978977b13f9e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiWebApiResourceConfig.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/NiFiWebApiResourceConfig.java @@ -104,6 +104,7 @@ public NiFiWebApiResourceConfig(@Context ServletContext servletContext) { register(ctx.getBean("versionsResource")); register(ctx.getBean("parameterContextResource")); register(ctx.getBean("parameterProviderResource")); + register(ctx.getBean("connectorResource")); // exception mappers register(AccessDeniedExceptionMapper.class); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java index 940310d3e8c0..bdf92b099056 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java @@ -52,6 +52,7 @@ import org.apache.nifi.authorization.user.NiFiUserUtils; import org.apache.nifi.bundle.Bundle; import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.FlowAnalysisRuleDefinition; import org.apache.nifi.c2.protocol.component.api.FlowRegistryClientDefinition; @@ -71,11 +72,19 @@ import org.apache.nifi.cluster.manager.exception.IllegalNodeDeletionException; import org.apache.nifi.cluster.manager.exception.UnknownNodeException; import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.components.ConfigVerificationResult; import org.apache.nifi.components.ConfigurableComponent; +import org.apache.nifi.components.DescribedValue; import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.RequiredPermission; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.components.connector.ConnectorUpdateContext; +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.secrets.AuthorizableSecret; import org.apache.nifi.components.state.Scope; import org.apache.nifi.components.state.StateMap; import org.apache.nifi.components.validation.ValidationState; @@ -162,7 +171,7 @@ import org.apache.nifi.parameter.ParameterLookup; import org.apache.nifi.parameter.ParameterProvider; import org.apache.nifi.parameter.ParameterReferenceManager; -import org.apache.nifi.parameter.StandardParameterContext; +import org.apache.nifi.parameter.StandardParameterContext.Builder; import org.apache.nifi.processor.Processor; import org.apache.nifi.processor.VerifiableProcessor; import org.apache.nifi.prometheusutil.AbstractMetricsRegistry; @@ -208,7 +217,7 @@ import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedPort; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedRemoteGroupPort; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.registry.flow.mapping.VersionedComponentStateLookup; import org.apache.nifi.remote.RemoteGroupPort; import org.apache.nifi.reporting.Bulletin; @@ -241,7 +250,9 @@ import org.apache.nifi.web.api.dto.ComponentValidationResultDTO; import org.apache.nifi.web.api.dto.ConfigVerificationResultDTO; import org.apache.nifi.web.api.dto.ConfigurationAnalysisDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; import org.apache.nifi.web.api.dto.ConnectionDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; import org.apache.nifi.web.api.dto.ControllerConfigurationDTO; import org.apache.nifi.web.api.dto.ControllerDTO; import org.apache.nifi.web.api.dto.ControllerServiceDTO; @@ -289,6 +300,7 @@ import org.apache.nifi.web.api.dto.RequiredPermissionDTO; import org.apache.nifi.web.api.dto.ResourceDTO; import org.apache.nifi.web.api.dto.RevisionDTO; +import org.apache.nifi.web.api.dto.SecretDTO; import org.apache.nifi.web.api.dto.SnippetDTO; import org.apache.nifi.web.api.dto.SystemDiagnosticsDTO; import org.apache.nifi.web.api.dto.TenantDTO; @@ -312,6 +324,7 @@ import org.apache.nifi.web.api.dto.search.SearchResultsDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatisticsDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; import org.apache.nifi.web.api.dto.status.ControllerStatusDTO; import org.apache.nifi.web.api.dto.status.NodeProcessGroupStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.PortStatusDTO; @@ -326,6 +339,7 @@ import org.apache.nifi.web.api.entity.ActionEntity; import org.apache.nifi.web.api.entity.ActivateControllerServicesEntity; import org.apache.nifi.web.api.entity.AffectedComponentEntity; +import org.apache.nifi.web.api.entity.AllowableValueEntity; import org.apache.nifi.web.api.entity.AssetEntity; import org.apache.nifi.web.api.entity.BulletinEntity; import org.apache.nifi.web.api.entity.ClearBulletinsForGroupResultsEntity; @@ -333,9 +347,13 @@ import org.apache.nifi.web.api.entity.ComponentReferenceEntity; import org.apache.nifi.web.api.entity.ComponentValidationResultEntity; import org.apache.nifi.web.api.entity.ConfigurationAnalysisEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; import org.apache.nifi.web.api.entity.ConnectionEntity; import org.apache.nifi.web.api.entity.ConnectionStatisticsEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; import org.apache.nifi.web.api.entity.ControllerBulletinsEntity; import org.apache.nifi.web.api.entity.ControllerConfigurationEntity; import org.apache.nifi.web.api.entity.ControllerServiceEntity; @@ -381,6 +399,7 @@ import org.apache.nifi.web.api.entity.RemoteProcessGroupStatusEntity; import org.apache.nifi.web.api.entity.ReportingTaskEntity; import org.apache.nifi.web.api.entity.ScheduleComponentsEntity; +import org.apache.nifi.web.api.entity.SecretsEntity; import org.apache.nifi.web.api.entity.SnippetEntity; import org.apache.nifi.web.api.entity.StartVersionControlRequestEntity; import org.apache.nifi.web.api.entity.StatusHistoryEntity; @@ -397,7 +416,9 @@ import org.apache.nifi.web.api.request.FlowMetricsReportingStrategy; import org.apache.nifi.web.controller.ControllerFacade; import org.apache.nifi.web.dao.AccessPolicyDAO; +import org.apache.nifi.web.dao.ComponentStateDAO; import org.apache.nifi.web.dao.ConnectionDAO; +import org.apache.nifi.web.dao.ConnectorDAO; import org.apache.nifi.web.dao.ControllerServiceDAO; import org.apache.nifi.web.dao.FlowAnalysisRuleDAO; import org.apache.nifi.web.dao.FlowRegistryDAO; @@ -448,6 +469,7 @@ import java.util.List; import java.util.ListIterator; import java.util.Map; +import java.util.Map.Entry; import java.util.Objects; import java.util.Optional; import java.util.Set; @@ -477,6 +499,7 @@ public class StandardNiFiServiceFacade implements NiFiServiceFacade { private BulletinRepository bulletinRepository; // data access objects + private ComponentStateDAO componentStateDAO; private ProcessorDAO processorDAO; private ProcessGroupDAO processGroupDAO; private RemoteProcessGroupDAO remoteProcessGroupDAO; @@ -488,6 +511,7 @@ public class StandardNiFiServiceFacade implements NiFiServiceFacade { private ConnectionDAO connectionDAO; private ControllerServiceDAO controllerServiceDAO; private ReportingTaskDAO reportingTaskDAO; + private ConnectorDAO connectorDAO; private FlowAnalysisRuleDAO flowAnalysisRuleDAO; private ParameterProviderDAO parameterProviderDAO; private UserDAO userDAO; @@ -778,6 +802,11 @@ public void verifyCreateReportingTask(ReportingTaskDTO reportingTaskDTO) { reportingTaskDAO.verifyCreate(reportingTaskDTO); } + @Override + public void verifyCreateConnector(final ConnectorDTO connectorDTO) { + connectorDAO.verifyCreate(connectorDTO); + } + @Override public void verifyUpdateReportingTask(final ReportingTaskDTO reportingTaskDTO) { // if tasks does not exist, then the update request is likely creating it @@ -1398,7 +1427,7 @@ private ParameterContextEntity createParameterContextEntity(final ParameterConte public List validateComponents(final ParameterContextDTO parameterContextDto, final NiFiUser nifiUser) { final ParameterContext parameterContext = parameterContextDAO.getParameterContext(parameterContextDto.getId()); final Set boundProcessGroups = parameterContext.getParameterReferenceManager().getProcessGroupsBound(parameterContext); - final ParameterContext updatedParameterContext = new StandardParameterContext.Builder() + final ParameterContext updatedParameterContext = new Builder() .id(parameterContext.getIdentifier()) .name(parameterContext.getName()) .parameterReferenceManager(ParameterReferenceManager.EMPTY) @@ -1736,7 +1765,7 @@ private void setEffectiveParameterUpdates(final ParameterContextDTO parameterCon .collect(Collectors.toMap(entity -> entity.getParameter().getName(), Function.identity())); parameterContextDto.getParameters().clear(); - for (final Map.Entry entry : proposedParameterUpdates.entrySet()) { + for (final Entry entry : proposedParameterUpdates.entrySet()) { final String parameterName = entry.getKey(); final Parameter parameter = entry.getValue(); final ParameterEntity parameterEntity; @@ -2066,6 +2095,68 @@ public ComponentStateDTO getRemoteProcessGroupState(String remoteProcessGroupId) return dtoFactory.createComponentStateDTO(remoteProcessGroupId, remoteProcessGroup.getClass(), localState, clusterState); } + @Override + public ComponentStateDTO getConnectorProcessorState(final String connectorId, final String processorId) { + final ProcessorNode processor = locateConnectorProcessor(connectorId, processorId); + final StateMap clusterState = isClustered() ? componentStateDAO.getState(processor, Scope.CLUSTER) : null; + final StateMap localState = componentStateDAO.getState(processor, Scope.LOCAL); + return dtoFactory.createComponentStateDTO(processorId, processor.getProcessor().getClass(), localState, clusterState); + } + + @Override + public void verifyCanClearConnectorProcessorState(final String connectorId, final String processorId) { + final ProcessorNode processor = locateConnectorProcessor(connectorId, processorId); + processor.verifyCanClearState(); + } + + @Override + public ComponentStateDTO clearConnectorProcessorState(final String connectorId, final String processorId, final ComponentStateDTO componentStateDTO) { + final ProcessorNode processor = locateConnectorProcessor(connectorId, processorId); + componentStateDAO.clearState(processor, componentStateDTO); + return getConnectorProcessorState(connectorId, processorId); + } + + @Override + public ComponentStateDTO getConnectorControllerServiceState(final String connectorId, final String controllerServiceId) { + final ControllerServiceNode controllerService = locateConnectorControllerService(connectorId, controllerServiceId); + final StateMap clusterState = isClustered() ? componentStateDAO.getState(controllerService, Scope.CLUSTER) : null; + final StateMap localState = componentStateDAO.getState(controllerService, Scope.LOCAL); + return dtoFactory.createComponentStateDTO(controllerServiceId, controllerService.getControllerServiceImplementation().getClass(), localState, clusterState); + } + + @Override + public void verifyCanClearConnectorControllerServiceState(final String connectorId, final String controllerServiceId) { + final ControllerServiceNode controllerService = locateConnectorControllerService(connectorId, controllerServiceId); + controllerService.verifyCanClearState(); + } + + @Override + public ComponentStateDTO clearConnectorControllerServiceState(final String connectorId, final String controllerServiceId, final ComponentStateDTO componentStateDTO) { + final ControllerServiceNode controllerService = locateConnectorControllerService(connectorId, controllerServiceId); + componentStateDAO.clearState(controllerService, componentStateDTO); + return getConnectorControllerServiceState(connectorId, controllerServiceId); + } + + private ProcessorNode locateConnectorProcessor(final String connectorId, final String processorId) { + final ConnectorNode connectorNode = connectorDAO.getConnector(connectorId); + final ProcessGroup managedGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ProcessorNode processor = managedGroup.findProcessor(processorId); + if (processor == null) { + throw new ResourceNotFoundException("Unable to find processor with id '%s' within connector '%s'.".formatted(processorId, connectorId)); + } + return processor; + } + + private ControllerServiceNode locateConnectorControllerService(final String connectorId, final String controllerServiceId) { + final ConnectorNode connectorNode = connectorDAO.getConnector(connectorId); + final ProcessGroup managedGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ControllerServiceNode controllerService = managedGroup.findControllerService(controllerServiceId, false, true); + if (controllerService == null) { + throw new ResourceNotFoundException("Unable to find controller service with id '%s' within connector '%s'.".formatted(controllerServiceId, connectorId)); + } + return controllerService; + } + @Override public ConnectionEntity deleteConnection(final Revision revision, final String connectionId) { final Connection connection = connectionDAO.getConnection(connectionId); @@ -2087,7 +2178,7 @@ public DropRequestDTO deleteFlowFileDropRequest(final String connectionId, final @Override public ListingRequestDTO deleteFlowFileListingRequest(final String connectionId, final String listingRequestId) { - final Connection connection = connectionDAO.getConnection(connectionId); + final Connection connection = connectionDAO.getConnection(connectionId, true); final ListingRequestDTO listRequest = dtoFactory.createListingRequestDTO(connectionDAO.deleteFlowFileListingRequest(connectionId, listingRequestId)); // include whether the source and destination are running @@ -2498,7 +2589,7 @@ public DropRequestDTO createFlowFileDropRequest(final String connectionId, final @Override public ListingRequestDTO createFlowFileListingRequest(final String connectionId, final String listingRequestId) { - final Connection connection = connectionDAO.getConnection(connectionId); + final Connection connection = connectionDAO.getConnection(connectionId, true); final ListingRequestDTO listRequest = dtoFactory.createListingRequestDTO(connectionDAO.createFlowFileListingRequest(connectionId, listingRequestId)); // include whether the source and destination are running @@ -3448,6 +3539,405 @@ public ReportingTaskEntity createReportingTask(final Revision revision, final Re return entityFactory.createReportingTaskEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, bulletinEntities); } + private ConnectorStatusDTO createConnectorStatusDto(final ConnectorNode connectorNode) { + final ProcessGroup managedProcessGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ProcessGroupStatus managedGroupStatus = controllerFacade.getProcessGroupStatus(managedProcessGroup.getIdentifier()); + return dtoFactory.createConnectorStatusDto(connectorNode, managedGroupStatus); + } + + @Override + public ConnectorEntity createConnector(final Revision revision, final ConnectorDTO connectorDTO) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + final BundleCoordinate coordinate = BundleUtils.getCompatibleBundle(controllerFacade.getExtensionManager(), connectorDTO.getType(), connectorDTO.getBundle()); + final ConnectorNode connector = connectorDAO.createConnector(connectorDTO.getType(), connectorDTO.getId(), coordinate, true, true); + + controllerFacade.save(); + + final ConnectorDTO dto = dtoFactory.createConnectorDto(connector); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode connector = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(connector); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(connector)); + final ConnectorStatusDTO status = createConnectorStatusDto(connector); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, status); + } + + @Override + public Set getConnectors() { + return connectorDAO.getConnectors().stream().map(node -> { + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final RevisionDTO revision = dtoFactory.createRevisionDTO(revisionManager.getRevision(node.getIdentifier())); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO status = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(dto, revision, permissions, operatePermissions, status); + }).collect(Collectors.toSet()); + } + + @Override + public ConnectorEntity getConnector(final String id) { + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final RevisionDTO revision = dtoFactory.createRevisionDTO(revisionManager.getRevision(node.getIdentifier())); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO status = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(dto, revision, permissions, operatePermissions, status); + } + + @Override + public void verifyUpdateConnector(final ConnectorDTO connectorDTO) { + // No-op placeholder for future detailed verification + } + + @Override + public ConnectorEntity updateConnector(final Revision revision, final ConnectorDTO connectorDTO) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + connectorDAO.updateConnector(connectorDTO); + + controllerFacade.save(); + + final ConnectorNode node = connectorDAO.getConnector(connectorDTO.getId()); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode node = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO statusDto = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, statusDto); + } + + @Override + public void verifyDeleteConnector(final String id) { + // For now, DAO will enforce state; expose hook for symmetry + } + + @Override + public ConnectorEntity deleteConnector(final Revision revision, final String id) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + connectorDAO.deleteConnector(id); + controllerFacade.save(); + + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorDTO dto = snapshot.getComponent(); + final PermissionsDTO permissions = new PermissionsDTO(); + permissions.setCanRead(Boolean.FALSE); + permissions.setCanWrite(Boolean.FALSE); + final PermissionsDTO operatePermissions = new PermissionsDTO(); + operatePermissions.setCanRead(Boolean.FALSE); + operatePermissions.setCanWrite(Boolean.FALSE); + return entityFactory.createConnectorEntity(dto, dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, null); + } + + @Override + public ConnectorEntity scheduleConnector(final Revision revision, final String id, final ScheduledState state) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + switch (state) { + case RUNNING -> connectorDAO.startConnector(id); + case STOPPED -> connectorDAO.stopConnector(id); + default -> throw new IllegalArgumentException("Unsupported scheduled state for Connector: " + state); + } + controllerFacade.save(); + + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode node = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO statusDto = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, statusDto); + } + + @Override + public void verifyDrainConnector(final String id) { + final ConnectorNode connector = connectorDAO.getConnector(id); + final ConnectorState currentState = connector.getCurrentState(); + if (currentState != ConnectorState.STOPPED) { + throw new IllegalStateException("Cannot drain FlowFiles for Connector " + id + " because it is not currently stopped. Current state: " + currentState); + } + } + + @Override + public ConnectorEntity drainConnector(final Revision revision, final String id) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + connectorDAO.drainFlowFiles(id); + controllerFacade.save(); + + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode node = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO statusDto = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, statusDto); + } + + @Override + public void verifyCancelConnectorDrain(final String id) { + connectorDAO.verifyCancelDrainFlowFile(id); + } + + @Override + public ConnectorEntity cancelConnectorDrain(final Revision revision, final String id) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + connectorDAO.cancelDrainFlowFiles(id); + controllerFacade.save(); + + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode node = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO statusDto = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, statusDto); + } + + @Override + public ConfigurationStepNamesEntity getConnectorConfigurationSteps(final String id) { + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + + return entityFactory.createConfigurationStepNamesEntity(dto); + } + + @Override + public ConfigurationStepEntity getConnectorConfigurationStep(final String id, final String configurationStepName) { + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final RevisionDTO revision = dtoFactory.createRevisionDTO(revisionManager.getRevision(node.getIdentifier())); + + return entityFactory.createConfigurationStepEntity(dto, configurationStepName, revision); + } + + @Override + public ConfigurationStepEntity updateConnectorConfigurationStep(final Revision revision, final String id, + final String configurationStepName, final ConfigurationStepConfigurationDTO configurationStepConfiguration) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + // Update the configuration step + connectorDAO.updateConnectorConfigurationStep(id, configurationStepName, configurationStepConfiguration); + controllerFacade.save(); + + // Return updated connector DTO + final ConnectorNode node = connectorDAO.getConnector(id); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + // Create and return the configuration step entity + return entityFactory.createConfigurationStepEntity(snapshot.getComponent(), configurationStepName, dtoFactory.createRevisionDTO(snapshot.getLastModification())); + } + + @Override + public ConnectorEntity applyConnectorUpdate(final Revision revision, final String connectorId) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final ConnectorUpdateContext updateContext = controllerFacade::saveImmediate; + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + connectorDAO.applyConnectorUpdate(connectorId, updateContext); + + final ConnectorNode node = connectorDAO.getConnector(connectorId); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode node = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO statusDto = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, statusDto); + } + + @Override + public ConnectorEntity discardConnectorUpdate(final Revision revision, final String connectorId) { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final RevisionClaim claim = new StandardRevisionClaim(revision); + + final RevisionUpdate snapshot = revisionManager.updateRevision(claim, user, () -> { + connectorDAO.discardWorkingConfiguration(connectorId); + + final ConnectorNode node = connectorDAO.getConnector(connectorId); + final ConnectorDTO dto = dtoFactory.createConnectorDto(node); + final FlowModification lastMod = new FlowModification(revision.incrementRevision(revision.getClientId()), user.getIdentity()); + return new StandardRevisionUpdate<>(dto, lastMod); + }); + + final ConnectorNode node = connectorDAO.getConnector(snapshot.getComponent().getId()); + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(node); + final PermissionsDTO operatePermissions = dtoFactory.createPermissionsDto(new OperationAuthorizable(node)); + final ConnectorStatusDTO statusDto = createConnectorStatusDto(node); + return entityFactory.createConnectorEntity(snapshot.getComponent(), dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions, operatePermissions, statusDto); + } + + @Override + public ProcessGroupFlowEntity getConnectorFlow(final String connectorId, final String processGroupId, final boolean uiOnly) { + final ConnectorNode connectorNode = connectorDAO.getConnector(connectorId); + final ProcessGroup managedProcessGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ProcessGroup targetProcessGroup = managedProcessGroup.findProcessGroup(processGroupId); + if (targetProcessGroup == null) { + throw new ResourceNotFoundException("Process Group with ID " + processGroupId + " was not found within Connector " + connectorId); + } + return createProcessGroupFlowEntity(targetProcessGroup, uiOnly); + } + + @Override + public ProcessGroupStatusEntity getConnectorProcessGroupStatus(final String id, final Boolean recursive) { + final ConnectorNode connectorNode = connectorDAO.getConnector(id); + final ProcessGroup managedProcessGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final String processGroupId = managedProcessGroup.getIdentifier(); + + final PermissionsDTO permissions = dtoFactory.createPermissionsDto(connectorNode); + final ProcessGroupStatusDTO dto = dtoFactory.createProcessGroupStatusDto(managedProcessGroup, controllerFacade.getProcessGroupStatus(processGroupId)); + + // prune the response as necessary + if (!Boolean.TRUE.equals(recursive)) { + pruneChildGroups(dto.getAggregateSnapshot()); + if (dto.getNodeSnapshots() != null) { + for (final NodeProcessGroupStatusSnapshotDTO nodeSnapshot : dto.getNodeSnapshots()) { + pruneChildGroups(nodeSnapshot.getStatusSnapshot()); + } + } + } + + return entityFactory.createProcessGroupStatusEntity(dto, permissions); + } + + @Override + public Set getConnectorControllerServices(final String connectorId, final String processGroupId, + final boolean includeAncestorGroups, final boolean includeDescendantGroups, final boolean includeReferencingComponents) { + final ConnectorNode connectorNode = connectorDAO.getConnector(connectorId); + final ProcessGroup managedProcessGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + final ProcessGroup targetProcessGroup = managedProcessGroup.findProcessGroup(processGroupId); + if (targetProcessGroup == null) { + throw new ResourceNotFoundException("Process Group with ID " + processGroupId + " was not found within Connector " + connectorId); + } + + final Set serviceNodes = new HashSet<>(); + serviceNodes.addAll(targetProcessGroup.getControllerServices(includeAncestorGroups)); + + if (includeDescendantGroups) { + serviceNodes.addAll(targetProcessGroup.findAllControllerServices()); + } + + return serviceNodes.stream() + .map(serviceNode -> createControllerServiceEntity(serviceNode, includeReferencingComponents)) + .collect(Collectors.toSet()); + } + + @Override + public void verifyCanVerifyConnectorConfigurationStep(final String connectorId, final String configurationStepName) { + connectorDAO.verifyCanVerifyConfigurationStep(connectorId, configurationStepName); + } + + @Override + public List performConnectorConfigurationStepVerification(final String connectorId, + final String configurationStepName, final ConfigurationStepConfigurationDTO configurationStepConfiguration) { + final List verificationResults = connectorDAO.verifyConfigurationStep(connectorId, configurationStepName, configurationStepConfiguration); + return verificationResults.stream() + .map(result -> dtoFactory.createConfigVerificationResultDto(result)) + .collect(Collectors.toList()); + } + + @Override + public SearchResultsDTO searchConnector(final String connectorId, final String query) { + final ConnectorNode connectorNode = connectorDAO.getConnector(connectorId); + final ProcessGroup managedProcessGroup = connectorNode.getActiveFlowContext().getManagedProcessGroup(); + return controllerFacade.searchConnector(query, managedProcessGroup); + } + + @Override + public ConnectorPropertyAllowableValuesEntity getConnectorPropertyAllowableValues(final String connectorId, + final String stepName, final String groupName, final String propertyName, final String filter) { + // groupName is retained for REST API backward compatibility but is no longer used by the underlying API + final List allowableValues = connectorDAO.fetchAllowableValues(connectorId, stepName, propertyName, filter); + + final List allowableValueEntities = allowableValues.stream() + .map(av -> entityFactory.createAllowableValueEntity(dtoFactory.createAllowableValueDto(av), true)) + .collect(Collectors.toList()); + + return entityFactory.createConnectorPropertyAllowableValuesEntity(stepName, groupName, propertyName, allowableValueEntities); + } + + @Override + public void verifyCreateConnectorAsset(final String connectorId) { + connectorDAO.verifyCreateAsset(connectorId); + } + + @Override + public AssetEntity createConnectorAsset(final String connectorId, final String assetId, final String assetName, final InputStream content) throws IOException { + final Asset createdAsset = connectorDAO.createAsset(connectorId, assetId, assetName, content); + return dtoFactory.createAssetEntity(createdAsset); + } + + @Override + public List getConnectorAssets(final String connectorId) { + return connectorDAO.getAssets(connectorId).stream() + .map(dtoFactory::createAssetEntity) + .toList(); + } + + @Override + public Optional getConnectorAsset(final String assetId) { + return connectorDAO.getAsset(assetId); + } + + @Override + public void verifyPurgeConnectorFlowFiles(final String connectorId) { + connectorDAO.verifyPurgeFlowFiles(connectorId); + } + + @Override + public void purgeConnectorFlowFiles(final String connectorId, final String requestor) { + connectorDAO.purgeFlowFiles(connectorId, requestor); + } + @Override public ReportingTaskEntity updateReportingTask(final Revision revision, final ReportingTaskDTO reportingTaskDTO) { // get the component, ensure we have access to it, and perform the update request @@ -3616,7 +4106,7 @@ public List getParameterContextUpdatesForAppliedParamete )); final Set updatedParameterEntities = new LinkedHashSet<>(); entity.getComponent().setParameters(updatedParameterEntities); - for (final Map.Entry parameterEntry : parameterUpdates.entrySet()) { + for (final Entry parameterEntry : parameterUpdates.entrySet()) { final String parameterName = parameterEntry.getKey(); final Parameter parameter = parameterEntry.getValue(); final ParameterEntity parameterEntity; @@ -3861,7 +4351,7 @@ public DropRequestDTO getFlowFileDropRequest(final String connectionId, final St @Override public ListingRequestDTO getFlowFileListingRequest(final String connectionId, final String listingRequestId) { - final Connection connection = connectionDAO.getConnection(connectionId); + final Connection connection = connectionDAO.getConnection(connectionId, true); final ListingRequestDTO listRequest = dtoFactory.createListingRequestDTO(connectionDAO.getFlowFileListingRequest(connectionId, listingRequestId)); // include whether the source and destination are running @@ -3964,6 +4454,11 @@ public Set getProcessorTypes(final String bundleGroup, final return controllerFacade.getFlowFileProcessorTypes(bundleGroup, bundleArtifact, type); } + @Override + public Set getConnectorTypes(final String bundleGroup, final String bundleArtifact, final String type) { + return controllerFacade.getConnectorTypes(bundleGroup, bundleArtifact, type); + } + @Override public Set getControllerServiceTypes(final String serviceType, final String serviceBundleGroup, final String serviceBundleArtifact, final String serviceBundleVersion, final String bundleGroup, final String bundleArtifact, final String type) { @@ -4039,11 +4534,25 @@ public FlowAnalysisRuleDefinition getFlowAnalysisRuleDefinition(String group, St return flowAnalysisRuleDefinition; } + @Override + public ConnectorDefinition getConnectorDefinition(final String group, final String artifact, final String version, final String type) { + final ConnectorDefinition connectorDefinition = controllerFacade.getConnectorDefinition(group, artifact, version, type); + if (connectorDefinition == null) { + throw new ResourceNotFoundException("Unable to find definition for [%s:%s:%s:%s]".formatted(group, artifact, version, type)); + } + return connectorDefinition; + } + @Override public String getAdditionalDetails(String group, String artifact, String version, String type) { return controllerFacade.getAdditionalDetails(group, artifact, version, type); } + @Override + public String getStepDocumentation(final String group, final String artifact, final String version, final String connectorType, final String stepName) { + return controllerFacade.getStepDocumentation(group, artifact, version, connectorType, stepName); + } + @Override public Set getParameterProviderTypes(final String bundleGroup, final String bundleArtifact, final String type) { return controllerFacade.getParameterProviderTypes(bundleGroup, bundleArtifact, type); @@ -4103,6 +4612,7 @@ private boolean authorizeBulletin(final Bulletin bulletin) { case OUTPUT_PORT -> authorizableLookup.getOutputPort(sourceId); case REMOTE_PROCESS_GROUP -> authorizableLookup.getRemoteProcessGroup(sourceId); case PROCESS_GROUP -> authorizableLookup.getProcessGroup(sourceId).getAuthorizable(); + case CONNECTOR -> authorizableLookup.getConnector(sourceId); default -> throw new IllegalArgumentException("Unexpected ComponentType: " + type); }; } catch (final ResourceNotFoundException e) { @@ -4951,6 +5461,7 @@ public CurrentUserEntity getCurrentUser() { entity.setPoliciesPermissions(dtoFactory.createPermissionsDto(authorizableLookup.getPolicies())); entity.setSystemPermissions(dtoFactory.createPermissionsDto(authorizableLookup.getSystem())); entity.setParameterContextPermissions(dtoFactory.createPermissionsDto(authorizableLookup.getParameterContexts())); + entity.setConnectorsPermissions(dtoFactory.createPermissionsDto(authorizableLookup.getConnectors())); entity.setCanVersionFlows(CollectionUtils.isNotEmpty(flowRegistryDAO.getFlowRegistryClients().stream().map(c -> c.getIdentifier()).collect(Collectors.toSet()))); entity.setRestrictedComponentsPermissions(dtoFactory.createPermissionsDto(authorizableLookup.getRestrictedComponents())); @@ -4977,12 +5488,15 @@ public CurrentUserEntity getCurrentUser() { @Override public ProcessGroupFlowEntity getProcessGroupFlow(final String groupId, final boolean uiOnly) { final ProcessGroup processGroup = processGroupDAO.getProcessGroup(groupId); + return createProcessGroupFlowEntity(processGroup, uiOnly); + } + private ProcessGroupFlowEntity createProcessGroupFlowEntity(final ProcessGroup processGroup, final boolean uiOnly) { // Get the Process Group Status but we only need a status depth of one because for any child process group, // we ignore the status of each individual components. I.e., if Process Group A has child Group B, and child Group B // has a Processor, we don't care about the individual stats of that Processor because the ProcessGroupFlowEntity // doesn't include that anyway. So we can avoid including the information in the status that is returned. - final ProcessGroupStatus groupStatus = controllerFacade.getProcessGroupStatus(groupId, 1); + final ProcessGroupStatus groupStatus = controllerFacade.getProcessGroupStatus(processGroup.getIdentifier(), 1); final RevisionDTO revision = dtoFactory.createRevisionDTO(revisionManager.getRevision(processGroup.getIdentifier())); final PermissionsDTO permissions = dtoFactory.createPermissionsDto(processGroup); return entityFactory.createProcessGroupFlowEntity(dtoFactory.createProcessGroupFlowDto(processGroup, groupStatus, @@ -5308,7 +5822,7 @@ public CopyResponseEntity copyComponents(final String groupId, final CopyRequest .mapAssetReferences(false) .build(); - final NiFiRegistryFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager(), mappingOptions); + final VersionedComponentFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager(), mappingOptions); final InstantiatedVersionedProcessGroup versionedProcessGroup = mapper.mapProcessGroup(processGroup, controllerFacade.getControllerServiceProvider(), controllerFacade.getFlowManager(), true); @@ -5341,7 +5855,7 @@ public CopyResponseEntity copyComponents(final String groupId, final CopyRequest final Map externalControllerServices = versionedProcessGroup.getExternalControllerServiceReferences().entrySet().stream() .filter(e -> isServiceReferenced(e.getKey(), versionedProcessors, Collections.emptySet(), versionedProcessGroups)) - .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); + .collect(Collectors.toMap(Entry::getKey, Entry::getValue)); // move any service at the current level into external services versionedProcessGroup.getControllerServices().stream() @@ -5417,7 +5931,7 @@ private RegisteredFlowSnapshot getCurrentFlowSnapshotByGroupId(final String proc // Create a complete (include descendant flows) VersionedProcessGroup snapshot of the flow as it is // currently without any registry related fields populated, even if the flow is currently versioned. - final NiFiRegistryFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); + final VersionedComponentFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); final InstantiatedVersionedProcessGroup nonVersionedProcessGroup = mapper.mapNonVersionedProcessGroup(processGroup, controllerFacade.getControllerServiceProvider()); @@ -5592,13 +6106,13 @@ public VersionControlInformationEntity getVersionControlInformation(final String private InstantiatedVersionedProcessGroup createFlowSnapshot(final String processGroupId) { final ProcessGroup processGroup = processGroupDAO.getProcessGroup(processGroupId); - final NiFiRegistryFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); + final VersionedComponentFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); final InstantiatedVersionedProcessGroup versionedGroup = mapper.mapProcessGroup(processGroup, controllerFacade.getControllerServiceProvider(), controllerFacade.getFlowManager(), false); return versionedGroup; } private Map createVersionedParameterContexts(final ProcessGroup processGroup, final Map parameterProviderReferences) { - final NiFiRegistryFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); + final VersionedComponentFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); return mapper.mapParameterContexts(processGroup, false, parameterProviderReferences); } @@ -5632,7 +6146,7 @@ public FlowComparisonEntity getLocalModifications(final String processGroupId) { throw new NiFiCoreException("Failed to retrieve flow with Flow Registry in order to calculate local differences due to " + e.getMessage(), e); } - final NiFiRegistryFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); + final VersionedComponentFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); final VersionedProcessGroup localGroup = mapper.mapProcessGroup(processGroup, controllerFacade.getControllerServiceProvider(), controllerFacade.getFlowManager(), true); final ComparableDataFlow localFlow = new StandardComparableDataFlow("Local Flow", localGroup); @@ -5796,7 +6310,7 @@ public void verifyCanRevertLocalModifications(final String groupId, final Regist public Set getComponentsAffectedByFlowUpdate(final String processGroupId, final RegisteredFlowSnapshot updatedSnapshot) { final ProcessGroup group = processGroupDAO.getProcessGroup(processGroupId); - final NiFiRegistryFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); + final VersionedComponentFlowMapper mapper = makeNiFiRegistryFlowMapper(controllerFacade.getExtensionManager()); final VersionedProcessGroup localContents = mapper.mapProcessGroup(group, controllerFacade.getControllerServiceProvider(), controllerFacade.getFlowManager(), true); final ComparableDataFlow localFlow = new StandardComparableDataFlow("Current Flow", localContents); @@ -6029,7 +6543,7 @@ private void mapToConnectableId(final Collection connecta if (versionedIdOption.isPresent()) { versionedId = versionedIdOption.get(); } else { - versionedId = NiFiRegistryFlowMapper.generateVersionedComponentId(connectable.getIdentifier()); + versionedId = VersionedComponentFlowMapper.generateVersionedComponentId(connectable.getIdentifier()); } final List byVersionedId = destination.computeIfAbsent(versionedId, key -> new ArrayList<>()); @@ -6455,6 +6969,7 @@ private AuthorizationResult authorizeAction(final Action action) { case AccessPolicy -> authorizableLookup.getAccessPolicyById(sourceId); case User, UserGroup -> authorizableLookup.getTenant(); case Label -> authorizableLookup.getLabel(sourceId); + case Connector -> authorizableLookup.getConnector(sourceId); default -> throw new IllegalArgumentException("Unexpected Component: " + type); }; } catch (final ResourceNotFoundException e) { @@ -6522,7 +7037,7 @@ public ComponentHistoryDTO getComponentHistory(final String componentId) { final Map propertyHistoryDtos = new LinkedHashMap<>(); final Map> propertyHistory = auditService.getPreviousValues(componentId); - for (final Map.Entry> entry : propertyHistory.entrySet()) { + for (final Entry> entry : propertyHistory.entrySet()) { final List previousValueDtos = new ArrayList<>(); for (final PreviousValue previousValue : entry.getValue()) { @@ -7044,6 +7559,7 @@ public NarDetailsEntity getNarDetails(final String identifier) { componentTypesEntity.setParameterProviderTypes(dtoFactory.fromDocumentedTypes(getTypes(extensionDefinitions, ParameterProvider.class))); componentTypesEntity.setFlowRegistryClientTypes(dtoFactory.fromDocumentedTypes(getTypes(extensionDefinitions, FlowRegistryClient.class))); componentTypesEntity.setFlowAnalysisRuleTypes(dtoFactory.fromDocumentedTypes(getTypes(extensionDefinitions, FlowAnalysisRule.class))); + componentTypesEntity.setConnectorTypes(dtoFactory.fromDocumentedTypes(getTypes(extensionDefinitions, Connector.class))); return componentTypesEntity; } @@ -7242,8 +7758,8 @@ public void verifyPublicOutputPortUniqueness(final String portId, final String p * @param extensionManager the extension manager to create the flow mapper with * @return a new NiFiRegistryFlowMapper instance */ - protected NiFiRegistryFlowMapper makeNiFiRegistryFlowMapper(final ExtensionManager extensionManager) { - return new NiFiRegistryFlowMapper(extensionManager); + protected VersionedComponentFlowMapper makeNiFiRegistryFlowMapper(final ExtensionManager extensionManager) { + return new VersionedComponentFlowMapper(extensionManager); } /** @@ -7253,8 +7769,8 @@ protected NiFiRegistryFlowMapper makeNiFiRegistryFlowMapper(final ExtensionManag * @param options the flow mapping options * @return a new NiFiRegistryFlowMapper instance */ - protected NiFiRegistryFlowMapper makeNiFiRegistryFlowMapper(final ExtensionManager extensionManager, final FlowMappingOptions options) { - return new NiFiRegistryFlowMapper(extensionManager, options); + protected VersionedComponentFlowMapper makeNiFiRegistryFlowMapper(final ExtensionManager extensionManager, final FlowMappingOptions options) { + return new VersionedComponentFlowMapper(extensionManager, options); } @Override @@ -7349,6 +7865,11 @@ public void setReportingTaskDAO(final ReportingTaskDAO reportingTaskDAO) { this.reportingTaskDAO = reportingTaskDAO; } + @Autowired + public void setConnectorDAO(final ConnectorDAO connectorDAO) { + this.connectorDAO = connectorDAO; + } + @Autowired public void setFlowAnalysisRuleDAO(FlowAnalysisRuleDAO flowAnalysisRuleDAO) { this.flowAnalysisRuleDAO = flowAnalysisRuleDAO; @@ -7359,6 +7880,11 @@ public void setParameterProviderDAO(final ParameterProviderDAO parameterProvider this.parameterProviderDAO = parameterProviderDAO; } + @Autowired + public void setComponentStateDAO(final ComponentStateDAO componentStateDAO) { + this.componentStateDAO = componentStateDAO; + } + @Autowired public void setParameterContextDAO(final ParameterContextDAO parameterContextDAO) { this.parameterContextDAO = parameterContextDAO; @@ -7489,4 +8015,23 @@ public Set filterComponents(final String groupId, final Function secrets = controllerFacade.getAllSecrets(); + final List secretDtos = secrets.stream() + .filter(secret -> isSecretAuthorized(secret, user)) + .map(dtoFactory::createSecretDto) + .toList(); + return entityFactory.createSecretsEntity(secretDtos); + } + + private boolean isSecretAuthorized(final Secret secret, final NiFiUser user) { + if (secret instanceof AuthorizableSecret authorizableSecret) { + final AuthorizationResult result = authorizableSecret.checkAuthorization(authorizer, RequestAction.READ, user); + return Result.Approved.equals(result.getResult()); + } + return true; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ConnectorResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ConnectorResource.java new file mode 100644 index 000000000000..c25cd0f450c7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ConnectorResource.java @@ -0,0 +1,2419 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api; + +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.Parameter; +import io.swagger.v3.oas.annotations.media.Content; +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.security.SecurityRequirement; +import io.swagger.v3.oas.annotations.tags.Tag; +import jakarta.servlet.ServletContext; +import jakarta.ws.rs.Consumes; +import jakarta.ws.rs.DELETE; +import jakarta.ws.rs.DefaultValue; +import jakarta.ws.rs.GET; +import jakarta.ws.rs.HeaderParam; +import jakarta.ws.rs.HttpMethod; +import jakarta.ws.rs.POST; +import jakarta.ws.rs.PUT; +import jakarta.ws.rs.Path; +import jakarta.ws.rs.PathParam; +import jakarta.ws.rs.Produces; +import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.core.Context; +import jakarta.ws.rs.core.HttpHeaders; +import jakarta.ws.rs.core.MediaType; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.StreamingOutput; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.asset.Asset; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.resource.DataAuthorizable; +import org.apache.nifi.authorization.resource.OperationAuthorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserUtils; +import org.apache.nifi.cluster.coordination.ClusterCoordinator; +import org.apache.nifi.cluster.coordination.http.replication.RequestReplicationHeader; +import org.apache.nifi.cluster.coordination.http.replication.UploadRequest; +import org.apache.nifi.cluster.coordination.http.replication.UploadRequestReplicator; +import org.apache.nifi.cluster.coordination.node.NodeConnectionState; +import org.apache.nifi.cluster.manager.NodeResponse; +import org.apache.nifi.cluster.protocol.NodeIdentifier; +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.stream.io.MaxLengthInputStream; +import org.apache.nifi.ui.extension.UiExtension; +import org.apache.nifi.ui.extension.UiExtensionMapping; +import org.apache.nifi.util.file.FileUtils; +import org.apache.nifi.web.NiFiServiceFacade; +import org.apache.nifi.web.ResourceNotFoundException; +import org.apache.nifi.web.Revision; +import org.apache.nifi.web.UiExtensionType; +import org.apache.nifi.web.api.concurrent.AsyncRequestManager; +import org.apache.nifi.web.api.concurrent.AsynchronousWebRequest; +import org.apache.nifi.web.api.concurrent.RequestManager; +import org.apache.nifi.web.api.concurrent.StandardAsynchronousWebRequest; +import org.apache.nifi.web.api.concurrent.StandardUpdateStep; +import org.apache.nifi.web.api.concurrent.UpdateStep; +import org.apache.nifi.web.api.dto.AssetDTO; +import org.apache.nifi.web.api.dto.BundleDTO; +import org.apache.nifi.web.api.dto.ComponentStateDTO; +import org.apache.nifi.web.api.dto.ConfigVerificationResultDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.DropRequestDTO; +import org.apache.nifi.web.api.dto.VerifyConnectorConfigStepRequestDTO; +import org.apache.nifi.web.api.dto.search.SearchResultsDTO; +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.AssetsEntity; +import org.apache.nifi.web.api.entity.ComponentStateEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; +import org.apache.nifi.web.api.entity.ConnectorRunStatusEntity; +import org.apache.nifi.web.api.entity.ControllerServiceEntity; +import org.apache.nifi.web.api.entity.ControllerServicesEntity; +import org.apache.nifi.web.api.entity.DropRequestEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessGroupStatusEntity; +import org.apache.nifi.web.api.entity.SearchResultsEntity; +import org.apache.nifi.web.api.entity.SecretsEntity; +import org.apache.nifi.web.api.entity.VerifyConnectorConfigStepRequestEntity; +import org.apache.nifi.web.api.request.ClientIdParameter; +import org.apache.nifi.web.api.request.LongParameter; +import org.apache.nifi.web.client.api.HttpResponseStatus; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Controller; + +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.util.Collections; +import java.util.Date; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; + +/** + * RESTful endpoint for managing a Connector. + */ +@Controller +@Path("/connectors") +@Tag(name = "Connectors") +public class ConnectorResource extends ApplicationResource { + + private static final Logger logger = LoggerFactory.getLogger(ConnectorResource.class); + private static final String VERIFICATION_REQUEST_TYPE = "verification-request"; + private static final String PURGE_REQUEST_TYPE = "purge-request"; + private static final String FILENAME_HEADER = "Filename"; + private static final String CONTENT_TYPE_HEADER = "Content-Type"; + private static final String UPLOAD_CONTENT_TYPE = "application/octet-stream"; + private static final long MAX_ASSET_SIZE_BYTES = (long) DataUnit.GB.toB(1); + + private NiFiServiceFacade serviceFacade; + private Authorizer authorizer; + private FlowResource flowResource; + private ControllerServiceResource controllerServiceResource; + private UploadRequestReplicator uploadRequestReplicator; + + private final RequestManager> configVerificationRequestManager = + new AsyncRequestManager<>(100, 1L, "Connector Configuration Step Verification"); + + private final RequestManager purgeRequestManager = new AsyncRequestManager<>(100, 1L, "Connector FlowFile Purge"); + + @Context + private ServletContext servletContext; + + /** + * Populate the uri's for the specified connectors. + * + * @param connectorEntities connectors + * @return connector entities + */ + public Set populateRemainingConnectorEntitiesContent(final Set connectorEntities) { + for (ConnectorEntity connectorEntity : connectorEntities) { + populateRemainingConnectorEntityContent(connectorEntity); + } + return connectorEntities; + } + + /** + * Populate the uri's for the specified connector. + * + * @param connectorEntity connector + * @return connector entity + */ + public ConnectorEntity populateRemainingConnectorEntityContent(final ConnectorEntity connectorEntity) { + connectorEntity.setUri(generateResourceUri("connectors", connectorEntity.getId())); + + // populate the remaining content + if (connectorEntity.getComponent() != null) { + populateRemainingConnectorContent(connectorEntity.getComponent()); + } + return connectorEntity; + } + + /** + * Populates the uri for the specified connector including custom UI information. + */ + public ConnectorDTO populateRemainingConnectorContent(final ConnectorDTO connector) { + final BundleDTO bundle = connector.getBundle(); + if (bundle == null) { + return connector; + } + + // see if this connector has any ui extensions + final UiExtensionMapping uiExtensionMapping = (UiExtensionMapping) servletContext.getAttribute("nifi-ui-extensions"); + if (uiExtensionMapping.hasUiExtension(connector.getType(), bundle.getGroup(), bundle.getArtifact(), bundle.getVersion())) { + final List uiExtensions = uiExtensionMapping.getUiExtension(connector.getType(), bundle.getGroup(), bundle.getArtifact(), bundle.getVersion()); + for (final UiExtension uiExtension : uiExtensions) { + if (UiExtensionType.Connector.equals(uiExtension.getExtensionType())) { + final String contextPath = uiExtension.getContextPath(); + final Map routes = uiExtension.getSupportedRoutes(); + + if (routes != null) { + final String configurationPath = routes.get("configuration"); + if (configurationPath != null) { + connector.setConfigurationUrl(buildCustomUiUrl(contextPath, configurationPath)); + } + + final String detailsPath = routes.get("details"); + if (detailsPath != null) { + connector.setDetailsUrl(buildCustomUiUrl(contextPath, detailsPath)); + } + } + } + } + } + + return connector; + } + + /** + * Builds a custom UI URL from the context path and route path. + * Handles both hash-based routing (path starts with #) and location-based routing. + * + * @param contextPath the context path of the custom UI + * @param routePath the route path (e.g., "#/wizard" for hash-based or "/wizard" for location-based) + * @return the full URL for the custom UI route + */ + private String buildCustomUiUrl(final String contextPath, final String routePath) { + final String baseUrl = generateExternalUiUri(contextPath); + if (routePath.startsWith("#")) { + // Hash-based routing: /context-path/#/route + return baseUrl + "/" + routePath; + } else { + // Location-based routing: /context-path/route + return baseUrl + routePath; + } + } + + /** + * Creates a new connector. + * + * @param requestConnectorEntity A connectorEntity. + * @return A connectorEntity. + */ + @POST + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Operation( + summary = "Creates a new connector", + responses = { + @ApiResponse(responseCode = "201", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors") + } + ) + public Response createConnector( + @Parameter(description = "The connector configuration details.", required = true) final ConnectorEntity requestConnectorEntity) { + + if (requestConnectorEntity == null || requestConnectorEntity.getComponent() == null) { + throw new IllegalArgumentException("Connector details must be specified."); + } + + if (requestConnectorEntity.getRevision() == null || (requestConnectorEntity.getRevision().getVersion() == null || requestConnectorEntity.getRevision().getVersion() != 0)) { + throw new IllegalArgumentException("A revision of 0 must be specified when creating a new Connector."); + } + + final ConnectorDTO requestConnector = requestConnectorEntity.getComponent(); + + if (StringUtils.isBlank(requestConnector.getType())) { + throw new IllegalArgumentException("The type of connector to create must be specified."); + } + + if (isReplicateRequest()) { + return replicate(HttpMethod.POST, requestConnectorEntity); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(requestConnectorEntity.isDisconnectedNodeAcknowledged()); + } + + return withWriteLock( + serviceFacade, + requestConnectorEntity, + lookup -> { + final Authorizable connectors = lookup.getConnectors(); + connectors.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> serviceFacade.verifyCreateConnector(requestConnector), + connectorEntity -> { + final ConnectorDTO connector = connectorEntity.getComponent(); + + // set the connector id as appropriate + final String clientSpecifiedId = connector.getId(); + if (clientSpecifiedId != null) { + // validate the client-specified connector id. uniqueness is verified by verifyCreateConnector() + connector.setId(normalizeUuid(clientSpecifiedId)); + } else { + connector.setId(generateUuid()); + } + + // create the new connector + final Revision revision = getRevision(connectorEntity, connector.getId()); + final ConnectorEntity entity = serviceFacade.createConnector(revision, connector); + populateRemainingConnectorEntityContent(entity); + + // generate a 201 created response + final String uri = entity.getUri(); + return generateCreatedResponse(URI.create(uri), entity).build(); + } + ); + } + + /** + * Validates client-specified id is a UUID and normalizes its string format to the lowercase digest used by NiFi. + * + * @param clientSpecifiedId the client-passed ID, which should be a valid UUID + * @return the normalized string representation of the validated UUID + * @throws IllegalArgumentException if the clientSpecifiedId is not a valid UUID + */ + private String normalizeUuid(final String clientSpecifiedId) { + try { + final UUID uuid = UUID.fromString(clientSpecifiedId); + logger.debug("ID [{}] is a valid UUID", clientSpecifiedId); + return uuid.toString(); + } catch (final Exception e) { + logger.error("ID [{}] is not a valid UUID", clientSpecifiedId, e); + throw new IllegalArgumentException("ID [" + clientSpecifiedId + "] is not a valid UUID."); + } + } + + /** + * Retrieves the specified connector. + * + * @param id The id of the connector to retrieve + * @return A connectorEntity. + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}") + @Operation( + summary = "Gets a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response getConnector( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // authorize access + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the connector + final ConnectorEntity entity = serviceFacade.getConnector(id); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + + /** + * Gets all available secrets from the SecretsManager for configuring a specific connector. + * + * @param id The id of the connector being configured + * @return A secretsEntity containing metadata for all available secrets. + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/secrets") + @Operation( + summary = "Gets all secrets available for configuring a connector", + description = "Returns metadata for all secrets available from all secret providers. " + + "This endpoint is used when configuring a connector to discover available secrets. " + + "Note: Actual secret values are not included in the response for security reasons.", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = SecretsEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response getSecrets( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + // NOTE: fetching secrets is handled by the node that receives the request and does not need to be replicated + // Secrets are sourced from ParameterProviders which should have consistent configuration across the cluster + + // authorize access - require write permission on the specific connector since this is used for configuration + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }); + + final SecretsEntity entity = serviceFacade.getSecrets(); + return generateOkResponse(entity).build(); + } + + /** + * Updates the specified connector. + * + * @param id The id of the connector to update. + * @param requestConnectorEntity A connectorEntity. + * @return A connectorEntity. + */ + @PUT + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}") + @Operation( + summary = "Updates a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response updateConnector( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The connector configuration details.", + required = true + ) final ConnectorEntity requestConnectorEntity) { + + if (requestConnectorEntity == null || requestConnectorEntity.getComponent() == null) { + throw new IllegalArgumentException("Connector details must be specified."); + } + + if (requestConnectorEntity.getRevision() == null) { + throw new IllegalArgumentException("Revision must be specified."); + } + + // ensure the ids are the same + final ConnectorDTO requestConnectorDTO = requestConnectorEntity.getComponent(); + if (!id.equals(requestConnectorDTO.getId())) { + throw new IllegalArgumentException(String.format("The connector id (%s) in the request body does not equal the " + + "connector id of the requested resource (%s).", requestConnectorDTO.getId(), id)); + } + + if (isReplicateRequest()) { + return replicate(HttpMethod.PUT, requestConnectorEntity); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(requestConnectorEntity.isDisconnectedNodeAcknowledged()); + } + + // handle expects request (usually from the cluster manager) + final Revision requestRevision = getRevision(requestConnectorEntity, id); + return withWriteLock( + serviceFacade, + requestConnectorEntity, + requestRevision, + lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> serviceFacade.verifyUpdateConnector(requestConnectorDTO), + (revision, connectorEntity) -> { + final ConnectorDTO connectorDTO = connectorEntity.getComponent(); + + // update the connector + final ConnectorEntity entity = serviceFacade.updateConnector(revision, connectorDTO); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Removes the specified connector. + * + * @param version The revision is used to verify the client is working with the latest version of the flow. + * @param clientId Optional client id. If the client id is not specified, a new one will be generated. This value (whether specified or generated) is included in the response. + * @param id The id of the connector to remove. + * @return A connectorEntity. + */ + @DELETE + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}") + @Operation( + summary = "Deletes a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response deleteConnector( + @Parameter( + description = "The revision is used to verify the client is working with the latest version of the flow." + ) + @QueryParam(VERSION) final LongParameter version, + @Parameter( + description = "If the client id is not specified, new one will be generated. This value (whether specified or generated) is included in the response." + ) + @QueryParam(CLIENT_ID) @DefaultValue(StringUtils.EMPTY) final ClientIdParameter clientId, + @Parameter( + description = "Acknowledges that this node is disconnected to allow for mutable requests to proceed." + ) + @QueryParam(DISCONNECTED_NODE_ACKNOWLEDGED) @DefaultValue("false") final Boolean disconnectedNodeAcknowledged, + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.DELETE); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(disconnectedNodeAcknowledged); + } + + final ConnectorEntity requestConnectorEntity = new ConnectorEntity(); + requestConnectorEntity.setId(id); + + // handle expects request (usually from the cluster manager) + final Revision requestRevision = new Revision(version == null ? null : version.getLong(), clientId.getClientId(), id); + return withWriteLock( + serviceFacade, + requestConnectorEntity, + requestRevision, + lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> serviceFacade.verifyDeleteConnector(id), + (revision, connectorEntity) -> { + // delete the specified connector + final ConnectorEntity entity = serviceFacade.deleteConnector(revision, connectorEntity.getId()); + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Updates run status of a connector. + * + * @param id The id of the connector to update. + * @param requestRunStatus A connectorRunStatusEntity. + * @return A connectorEntity. + */ + @PUT + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/run-status") + @Operation( + summary = "Updates run status of a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid} or /operation/connectors/{uuid}") + } + ) + public Response updateRunStatus( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The connector run status.", + required = true + ) final ConnectorRunStatusEntity requestRunStatus) { + + if (requestRunStatus == null) { + throw new IllegalArgumentException("Connector run status must be specified."); + } + + if (requestRunStatus.getRevision() == null) { + throw new IllegalArgumentException("Revision must be specified."); + } + + requestRunStatus.validateState(); + + if (isReplicateRequest()) { + return replicate(HttpMethod.PUT, requestRunStatus); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(requestRunStatus.isDisconnectedNodeAcknowledged()); + } + + // handle expects request (usually from the cluster manager) + final Revision requestRevision = getRevision(requestRunStatus.getRevision(), id); + return withWriteLock( + serviceFacade, + requestRunStatus, + requestRevision, + lookup -> { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + + final Authorizable connector = lookup.getConnector(id); + OperationAuthorizable.authorizeOperation(connector, authorizer, user); + }, + () -> serviceFacade.verifyUpdateConnector(createDTOWithDesiredRunStatus(id, requestRunStatus.getState())), + (revision, runStatusEntity) -> { + // update the connector + final ScheduledState scheduledState = ScheduledState.valueOf(runStatusEntity.getState()); + final ConnectorEntity entity = serviceFacade.scheduleConnector(revision, id, scheduledState); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Initiates the draining of FlowFiles for the specified connector. + * + * @param id The id of the connector to drain. + * @param requestConnectorEntity A connectorEntity containing the revision. + * @return A connectorEntity. + */ + @POST + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/drain") + @Operation( + summary = "Initiates draining of FlowFiles for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "This will initiate draining of FlowFiles for a stopped connector. Draining allows the connector to process " + + "data that is currently in the flow but does not ingest any additional data. The connector must be in a STOPPED state " + + "before draining can begin. Once initiated, the connector will transition to a DRAINING state. Use the DELETE method " + + "on this endpoint to cancel an ongoing drain operation.", + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid} or /operation/connectors/{uuid}") + } + ) + public Response initiateDrain( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The connector entity with revision.", + required = true + ) final ConnectorEntity requestConnectorEntity) { + + if (requestConnectorEntity == null || requestConnectorEntity.getRevision() == null) { + throw new IllegalArgumentException("Connector entity with revision must be specified."); + } + + if (requestConnectorEntity.getId() != null && !id.equals(requestConnectorEntity.getId())) { + throw new IllegalArgumentException(String.format("The connector id (%s) in the request body does not equal the " + + "connector id of the requested resource (%s).", requestConnectorEntity.getId(), id)); + } + + if (isReplicateRequest()) { + return replicate(HttpMethod.POST, requestConnectorEntity); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(requestConnectorEntity.isDisconnectedNodeAcknowledged()); + } + + final Revision requestRevision = getRevision(requestConnectorEntity, id); + return withWriteLock( + serviceFacade, + requestConnectorEntity, + requestRevision, + lookup -> { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final Authorizable connector = lookup.getConnector(id); + OperationAuthorizable.authorizeOperation(connector, authorizer, user); + }, + () -> serviceFacade.verifyDrainConnector(id), + (revision, connectorEntity) -> { + final ConnectorEntity entity = serviceFacade.drainConnector(revision, id); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Cancels the draining of FlowFiles for the specified connector. + * + * @param version The revision is used to verify the client is working with the latest version of the flow. + * @param clientId Optional client id. If the client id is not specified, a new one will be generated. This value (whether specified or generated) is included in the response. + * @param id The id of the connector to cancel draining for. + * @return A connectorEntity. + */ + @DELETE + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/drain") + @Operation( + summary = "Cancels the draining of FlowFiles for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid} or /operation/connectors/{uuid}") + } + ) + public Response cancelDrain( + @Parameter( + description = "The revision is used to verify the client is working with the latest version of the flow." + ) + @QueryParam(VERSION) final LongParameter version, + @Parameter( + description = "If the client id is not specified, new one will be generated. This value (whether specified or generated) is included in the response." + ) + @QueryParam(CLIENT_ID) @DefaultValue(StringUtils.EMPTY) final ClientIdParameter clientId, + @Parameter( + description = "Acknowledges that this node is disconnected to allow for mutable requests to proceed." + ) + @QueryParam(DISCONNECTED_NODE_ACKNOWLEDGED) @DefaultValue("false") final Boolean disconnectedNodeAcknowledged, + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.DELETE); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(disconnectedNodeAcknowledged); + } + + final ConnectorEntity requestConnectorEntity = new ConnectorEntity(); + requestConnectorEntity.setId(id); + + final Revision requestRevision = new Revision(version == null ? null : version.getLong(), clientId.getClientId(), id); + return withWriteLock( + serviceFacade, + requestConnectorEntity, + requestRevision, + lookup -> { + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final Authorizable connector = lookup.getConnector(id); + OperationAuthorizable.authorizeOperation(connector, authorizer, user); + }, + () -> serviceFacade.verifyCancelConnectorDrain(id), + (revision, connectorEntity) -> { + final ConnectorEntity entity = serviceFacade.cancelConnectorDrain(revision, id); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + ); + } + + + @POST + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/purge-requests") + @Operation( + summary = "Creates a request to purge the FlowFiles for this connector", + responses = { + @ApiResponse( + responseCode = "202", description = "The request has been accepted. A HTTP response header will contain the URI where the response can be polled.", + content = @Content(schema = @Schema(implementation = DropRequestEntity.class)) + ), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "This will create a request to purge all FlowFiles from the connector. The connector must be in a STOPPED state before purging can begin. " + + "This is an asynchronous operation. The client should poll the returned URI to get the status of the purge request.", + security = { + @SecurityRequirement(name = "Write Source Data - /data/connectors/{uuid}") + } + ) + public Response createPurgeRequest( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.POST); + } + + final ConnectorEntity requestConnectorEntity = new ConnectorEntity(); + requestConnectorEntity.setId(id); + + return withWriteLock( + serviceFacade, + requestConnectorEntity, + lookup -> { + final Authorizable dataAuthorizable = new DataAuthorizable(lookup.getConnector(id)); + dataAuthorizable.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> serviceFacade.verifyPurgeConnectorFlowFiles(id), + (connectorEntity) -> performAsyncPurge(connectorEntity, id, NiFiUserUtils.getNiFiUser()) + ); + } + + + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/purge-requests/{purge-request-id}") + @Operation( + summary = "Gets the current status of a purge request for the specified connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = DropRequestEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Only the user that submitted the request can get it") + } + ) + public Response getPurgeRequest( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String connectorId, + @Parameter( + description = "The purge request id.", + required = true + ) + @PathParam("purge-request-id") final String purgeRequestId) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + serviceFacade.authorizeAccess(lookup -> { + final Authorizable dataAuthorizable = new DataAuthorizable(lookup.getConnector(connectorId)); + dataAuthorizable.authorize(authorizer, RequestAction.WRITE, user); + }); + + final AsynchronousWebRequest asyncRequest = purgeRequestManager.getRequest(PURGE_REQUEST_TYPE, purgeRequestId, user); + final DropRequestEntity purgeRequestEntity = createPurgeRequestEntity(asyncRequest, connectorId, purgeRequestId); + return generateOkResponse(purgeRequestEntity).build(); + } + + + @DELETE + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/purge-requests/{purge-request-id}") + @Operation( + summary = "Cancels and/or removes a request to purge the FlowFiles for this connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = DropRequestEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Only the user that submitted the request can remove it") + } + ) + public Response removePurgeRequest( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String connectorId, + @Parameter( + description = "The purge request id.", + required = true + ) + @PathParam("purge-request-id") final String purgeRequestId) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.DELETE); + } + + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + + // Make sure user has write access to the connector data + serviceFacade.authorizeAccess(lookup -> { + final Authorizable dataAuthorizable = new DataAuthorizable(lookup.getConnector(connectorId)); + dataAuthorizable.authorize(authorizer, RequestAction.WRITE, user); + }); + + final boolean twoPhaseRequest = isTwoPhaseRequest(httpServletRequest); + final boolean executionPhase = isExecutionPhase(httpServletRequest); + + if (!twoPhaseRequest || executionPhase) { + final AsynchronousWebRequest asyncRequest = purgeRequestManager.removeRequest(PURGE_REQUEST_TYPE, purgeRequestId, user); + + if (!asyncRequest.isComplete()) { + asyncRequest.cancel(); + } + + final DropRequestEntity purgeRequestEntity = createPurgeRequestEntity(asyncRequest, connectorId, purgeRequestId); + return generateOkResponse(purgeRequestEntity).build(); + } + + if (isValidationPhase(httpServletRequest)) { + purgeRequestManager.getRequest(PURGE_REQUEST_TYPE, purgeRequestId, user); + return generateContinueResponse().build(); + } else if (isCancellationPhase(httpServletRequest)) { + return generateOkResponse().build(); + } else { + throw new IllegalStateException("This request does not appear to be part of the two phase commit."); + } + } + + private Response performAsyncPurge(final ConnectorEntity connectorEntity, final String connectorId, final NiFiUser user) { + final String requestId = generateUuid(); + logger.debug("Generated Purge Request with ID {} for Connector {}", requestId, connectorId); + + final List updateSteps = Collections.singletonList(new StandardUpdateStep("Purge FlowFiles")); + + final AsynchronousWebRequest request = + new StandardAsynchronousWebRequest<>(requestId, connectorEntity, connectorId, user, updateSteps); + + final Consumer> updateTask = asyncRequest -> { + try { + serviceFacade.purgeConnectorFlowFiles(connectorId, user.getIdentity()); + asyncRequest.markStepComplete(null); + } catch (final Exception e) { + logger.error("Failed to purge FlowFiles for Connector {}", connectorId, e); + asyncRequest.fail("Failed to purge FlowFiles due to " + e); + } + }; + + purgeRequestManager.submitRequest(PURGE_REQUEST_TYPE, requestId, request, updateTask); + + final DropRequestEntity purgeRequestEntity = createPurgeRequestEntity(request, connectorId, requestId); + final URI location = URI.create(purgeRequestEntity.getDropRequest().getUri()); + return Response.status(Response.Status.ACCEPTED).location(location).entity(purgeRequestEntity).build(); + } + + private DropRequestEntity createPurgeRequestEntity(final AsynchronousWebRequest asyncRequest, + final String connectorId, final String requestId) { + final DropRequestDTO dto = new DropRequestDTO(); + dto.setId(requestId); + dto.setUri(generateResourceUri("connectors", connectorId, "purge-requests", requestId)); + dto.setSubmissionTime(asyncRequest.getLastUpdated()); + dto.setLastUpdated(asyncRequest.getLastUpdated()); + dto.setPercentCompleted(asyncRequest.getPercentComplete()); + dto.setFinished(asyncRequest.isComplete()); + dto.setFailureReason(asyncRequest.getFailureReason()); + dto.setState(asyncRequest.getState()); + + final DropRequestEntity entity = new DropRequestEntity(); + entity.setDropRequest(dto); + return entity; + } + + /** + * Gets the configuration step names for the specified connector. + * + * @param id The id of the connector to retrieve configuration steps from + * @return A configurationStepNamesEntity. + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps") + @Operation( + summary = "Gets all configuration step names for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConfigurationStepNamesEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response getConnectorConfigurationSteps( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // authorize access + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the configuration step names + final ConfigurationStepNamesEntity entity = serviceFacade.getConnectorConfigurationSteps(id); + + return generateOkResponse(entity).build(); + } + + /** + * Gets a specific configuration step by name for the specified connector. + * + * @param id The id of the connector to retrieve configuration step from + * @param configurationStepName The name of the configuration step to retrieve + * @return A configurationStepEntity. + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps/{configurationStepName}") + @Operation( + summary = "Gets a specific configuration step by name for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConfigurationStepEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response getConnectorConfigurationStep( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The configuration step name.", + required = true + ) + @PathParam("configurationStepName") final String configurationStepName) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // authorize access + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the specific configuration step + final ConfigurationStepEntity entity = serviceFacade.getConnectorConfigurationStep(id, configurationStepName); + + return generateOkResponse(entity).build(); + } + + /** + * Gets the allowable values for a specific property in a connector's configuration step. + * + * @param id The id of the connector + * @param configurationStepName The name of the configuration step + * @param propertyGroupName The name of the property group + * @param propertyName The name of the property + * @param filter Optional filter for the allowable values + * @return A ConnectorPropertyAllowableValuesEntity + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps/{configurationStepName}/property-groups/{propertyGroupName}/properties/{propertyName}/allowable-values") + @Operation( + summary = "Gets the allowable values for a specific property in a connector's configuration step", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorPropertyAllowableValuesEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "Gets the allowable values for a specific property that supports dynamic fetching of allowable values. " + + "The filter parameter can be used to narrow down the results based on the property's filtering logic.", + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response getConnectorPropertyAllowableValues( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String id, + @Parameter(description = "The configuration step name.", required = true) + @PathParam("configurationStepName") final String configurationStepName, + @Parameter(description = "The property group name.", required = true) + @PathParam("propertyGroupName") final String propertyGroupName, + @Parameter(description = "The property name.", required = true) + @PathParam("propertyName") final String propertyName, + @Parameter(description = "Optional filter to narrow down the allowable values.") + @QueryParam("filter") final String filter) { + + // NOTE: fetching allowable values is handled by the node that receives the request and does not need to be replicated + + // authorize access + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the allowable values + final ConnectorPropertyAllowableValuesEntity entity = serviceFacade.getConnectorPropertyAllowableValues(id, configurationStepName, propertyGroupName, propertyName, filter); + + return generateOkResponse(entity).build(); + } + + /** + * Updates a specific configuration step by name for the specified connector. + * + * @param id The id of the connector to update configuration step for + * @param configurationStepName The name of the configuration step to update + * @param requestConfigurationStepEntity The configuration step configuration to update + * @return A configurationStepEntity. + */ + @PUT + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps/{configurationStepName}") + @Operation( + summary = "Updates a specific configuration step by name for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConfigurationStepEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response updateConnectorConfigurationStep( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The configuration step name.", + required = true + ) + @PathParam("configurationStepName") final String configurationStepName, + @Parameter( + description = "The configuration step configuration.", + required = true + ) final ConfigurationStepEntity requestConfigurationStepEntity) { + + if (requestConfigurationStepEntity == null || requestConfigurationStepEntity.getConfigurationStep() == null) { + throw new IllegalArgumentException("Configuration step details must be specified."); + } + + if (requestConfigurationStepEntity.getParentConnectorRevision() == null) { + throw new IllegalArgumentException("Parent connector revision must be specified."); + } + + if (requestConfigurationStepEntity.getParentConnectorId() == null) { + throw new IllegalArgumentException("Parent connector ID must be specified."); + } + + if (!id.equals(requestConfigurationStepEntity.getParentConnectorId())) { + throw new IllegalArgumentException(String.format("The parent connector ID (%s) in the request body does not equal the " + + "connector ID of the requested resource (%s).", requestConfigurationStepEntity.getParentConnectorId(), id)); + } + + // ensure the configuration step names match + final ConfigurationStepConfigurationDTO requestConfigurationStep = requestConfigurationStepEntity.getConfigurationStep(); + if (!configurationStepName.equals(requestConfigurationStep.getConfigurationStepName())) { + throw new IllegalArgumentException(String.format("The configuration step name (%s) in the request body does not equal the " + + "configuration step name of the requested resource (%s).", requestConfigurationStep.getConfigurationStepName(), configurationStepName)); + } + + if (isReplicateRequest()) { + return replicate(HttpMethod.PUT, requestConfigurationStepEntity); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(requestConfigurationStepEntity.isDisconnectedNodeAcknowledged()); + } + + // handle expects request (usually from the cluster manager) + final Revision requestRevision = getRevision(requestConfigurationStepEntity.getParentConnectorRevision(), id); + return withWriteLock( + serviceFacade, + requestConfigurationStepEntity, + requestRevision, + lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> { + // Verify the connector exists and the configuration step exists + serviceFacade.getConnectorConfigurationStep(id, configurationStepName); + }, + (revision, configurationStepEntity) -> { + final ConfigurationStepConfigurationDTO configurationStepConfiguration = configurationStepEntity.getConfigurationStep(); + + // update the configuration step + final ConfigurationStepEntity entity = serviceFacade.updateConnectorConfigurationStep(revision, id, configurationStepName, configurationStepConfiguration); + + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Submits a request to perform verification of a specific configuration step for a connector. + * This is an asynchronous operation that will return immediately with a request ID that can be + * used to poll for the results. + * + * @param id The id of the connector + * @param configurationStepName The name of the configuration step to verify + * @param requestEntity The verify config request entity containing the properties to verify + * @return The verification request entity with the request ID + */ + @POST + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps/{configurationStepName}/verify-config") + @Operation( + summary = "Performs verification of a configuration step for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = VerifyConnectorConfigStepRequestEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "This will initiate the process of verifying a given Connector Configuration Step. This may be a long-running task. As a result, " + + "this endpoint will immediately return a VerifyConnectorConfigStepRequestEntity, and the process of performing the verification will occur asynchronously in the background. " + + "The client may then periodically poll the status of the request by issuing a GET request to " + + "/connectors/{connectorId}/configuration-steps/{stepName}/verify-config/{requestId}. Once the request is completed, the client is expected to issue a DELETE request to " + + "/connectors/{connectorId}/configuration-steps/{stepName}/verify-config/{requestId}.", + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response submitConfigurationStepVerificationRequest( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String id, + @Parameter(description = "The configuration step name.", required = true) + @PathParam("configurationStepName") final String configurationStepName, + @Parameter(description = "The verify config request entity containing the configuration step to verify.", required = true) + final VerifyConnectorConfigStepRequestEntity requestEntity) { + + if (requestEntity == null) { + throw new IllegalArgumentException("Connector configuration step verification request must be specified."); + } + + final VerifyConnectorConfigStepRequestDTO requestDto = requestEntity.getRequest(); + if (requestDto == null || requestDto.getConfigurationStep() == null) { + throw new IllegalArgumentException("Connector configuration step must be specified."); + } + + if (requestDto.getConnectorId() == null) { + throw new IllegalArgumentException("Connector's identifier must be specified in the request."); + } + + if (!requestDto.getConnectorId().equals(id)) { + throw new IllegalArgumentException("Connector's identifier in the request must match the identifier provided in the URL."); + } + + if (requestDto.getConfigurationStepName() == null) { + throw new IllegalArgumentException("Configuration step name must be specified in the request."); + } + + if (!requestDto.getConfigurationStepName().equals(configurationStepName)) { + throw new IllegalArgumentException("Configuration step name in the request must match the step name provided in the URL."); + } + + if (isReplicateRequest()) { + return replicate(HttpMethod.POST, requestEntity); + } + + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + + return withWriteLock( + serviceFacade, + requestEntity, + lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> { + serviceFacade.verifyCanVerifyConnectorConfigurationStep(id, configurationStepName); + }, + entity -> performAsyncConfigStepVerification(entity, id, configurationStepName, user) + ); + } + + /** + * Returns the verification request with the given ID for a connector configuration step. + * + * @param id The id of the connector + * @param configurationStepName The name of the configuration step + * @param requestId The id of the verification request + * @return The verification request entity + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps/{configurationStepName}/verify-config/{requestId}") + @Operation( + summary = "Returns the Verification Request with the given ID", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = VerifyConnectorConfigStepRequestEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "Returns the Verification Request with the given ID. Once a Verification Request has been created, " + + "that request can subsequently be retrieved via this endpoint, and the request that is fetched will contain the updated state, such as percent complete, the " + + "current state of the request, and any failures.", + security = { + @SecurityRequirement(name = "Only the user that submitted the request can get it") + } + ) + public Response getConfigurationStepVerificationRequest( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String id, + @Parameter(description = "The configuration step name.", required = true) + @PathParam("configurationStepName") final String configurationStepName, + @Parameter(description = "The ID of the Verification Request", required = true) + @PathParam("requestId") final String requestId) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + + final AsynchronousWebRequest> asyncRequest = + configVerificationRequestManager.getRequest(VERIFICATION_REQUEST_TYPE, requestId, user); + final VerifyConnectorConfigStepRequestEntity updateRequestEntity = createVerifyConnectorConfigStepRequestEntity(asyncRequest, id, configurationStepName, requestId); + return generateOkResponse(updateRequestEntity).build(); + } + + /** + * Deletes the verification request with the given ID for a connector configuration step. + * + * @param id The id of the connector + * @param configurationStepName The name of the configuration step + * @param requestId The id of the verification request + * @return The verification request entity + */ + @DELETE + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/configuration-steps/{configurationStepName}/verify-config/{requestId}") + @Operation( + summary = "Deletes the Verification Request with the given ID", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = VerifyConnectorConfigStepRequestEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "Deletes the Verification Request with the given ID. After a request is created, it is expected " + + "that the client will properly clean up the request by DELETE'ing it, once the Verification process has completed. If the request is deleted before the request " + + "completes, then the Verification request will finish the step that it is currently performing and then will cancel any subsequent steps.", + security = { + @SecurityRequirement(name = "Only the user that submitted the request can remove it") + } + ) + public Response deleteConfigurationStepVerificationRequest( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String id, + @Parameter(description = "The configuration step name.", required = true) + @PathParam("configurationStepName") final String configurationStepName, + @Parameter(description = "The ID of the Verification Request", required = true) + @PathParam("requestId") final String requestId) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.DELETE); + } + + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + final boolean twoPhaseRequest = isTwoPhaseRequest(httpServletRequest); + final boolean executionPhase = isExecutionPhase(httpServletRequest); + + if (!twoPhaseRequest || executionPhase) { + final AsynchronousWebRequest> asyncRequest = + configVerificationRequestManager.removeRequest(VERIFICATION_REQUEST_TYPE, requestId, user); + + if (!asyncRequest.isComplete()) { + asyncRequest.cancel(); + } + + final VerifyConnectorConfigStepRequestEntity updateRequestEntity = createVerifyConnectorConfigStepRequestEntity(asyncRequest, id, configurationStepName, requestId); + return generateOkResponse(updateRequestEntity).build(); + } + + if (isValidationPhase(httpServletRequest)) { + configVerificationRequestManager.getRequest(VERIFICATION_REQUEST_TYPE, requestId, user); + return generateContinueResponse().build(); + } else if (isCancellationPhase(httpServletRequest)) { + return generateOkResponse().build(); + } else { + throw new IllegalStateException("This request does not appear to be part of the two phase commit."); + } + } + + private Response performAsyncConfigStepVerification(final VerifyConnectorConfigStepRequestEntity requestEntity, final String connectorId, + final String configurationStepName, final NiFiUser user) { + final String requestId = generateUuid(); + logger.debug("Generated Config Verification Request with ID {} for Connector {} Configuration Step {}", requestId, connectorId, configurationStepName); + + final VerifyConnectorConfigStepRequestDTO requestDto = requestEntity.getRequest(); + final List updateSteps = Collections.singletonList(new StandardUpdateStep("Verify Connector Configuration Step")); + + final AsynchronousWebRequest> request = + new StandardAsynchronousWebRequest<>(requestId, requestEntity, connectorId, user, updateSteps); + + final Consumer>> updateTask = asyncRequest -> { + try { + final ConfigurationStepConfigurationDTO configurationStep = requestDto.getConfigurationStep(); + final List results = serviceFacade.performConnectorConfigurationStepVerification(connectorId, configurationStepName, configurationStep); + asyncRequest.markStepComplete(results); + } catch (final Exception e) { + logger.error("Failed to verify Connector Configuration Step", e); + asyncRequest.fail("Failed to verify Connector Configuration Step due to " + e); + } + }; + + configVerificationRequestManager.submitRequest(VERIFICATION_REQUEST_TYPE, requestId, request, updateTask); + + final VerifyConnectorConfigStepRequestEntity resultsEntity = createVerifyConnectorConfigStepRequestEntity(request, connectorId, configurationStepName, requestId); + return generateOkResponse(resultsEntity).build(); + } + + private VerifyConnectorConfigStepRequestEntity createVerifyConnectorConfigStepRequestEntity( + final AsynchronousWebRequest> asyncRequest, + final String connectorId, final String configurationStepName, final String requestId) { + + final VerifyConnectorConfigStepRequestDTO requestDto = asyncRequest.getRequest().getRequest(); + final List resultsList = asyncRequest.getResults(); + + final VerifyConnectorConfigStepRequestDTO dto = new VerifyConnectorConfigStepRequestDTO(); + dto.setConnectorId(requestDto.getConnectorId()); + dto.setConfigurationStepName(requestDto.getConfigurationStepName()); + dto.setConfigurationStep(requestDto.getConfigurationStep()); + dto.setResults(resultsList); + + dto.setComplete(asyncRequest.isComplete()); + dto.setFailureReason(asyncRequest.getFailureReason()); + dto.setLastUpdated(asyncRequest.getLastUpdated()); + dto.setPercentCompleted(asyncRequest.getPercentComplete()); + dto.setRequestId(requestId); + dto.setState(asyncRequest.getState()); + dto.setUri(generateResourceUri("connectors", connectorId, "configuration-steps", configurationStepName, "verify-config", requestId)); + + final VerifyConnectorConfigStepRequestEntity entity = new VerifyConnectorConfigStepRequestEntity(); + entity.setRequest(dto); + return entity; + } + + /** + * Performs a search request within the encapsulated process group of this connector. + * + * @param id The connector id + * @param value Search string + * @return A searchResultsEntity + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/search-results") + @Operation( + summary = "Performs a search against the encapsulated process group of this connector using the specified search term", + description = "Only search results from authorized components will be returned.", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = SearchResultsEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response searchConnector( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The search term.", + required = false + ) + @QueryParam("q") @DefaultValue(StringUtils.EMPTY) final String value + ) { + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // authorize access to the connector + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // search within the connector's encapsulated process group + final SearchResultsDTO results = serviceFacade.searchConnector(id, value); + + // create the entity + final SearchResultsEntity entity = new SearchResultsEntity(); + entity.setSearchResultsDTO(results); + + // generate the response + return noCache(Response.ok(entity)).build(); + } + + private ConnectorDTO createDTOWithDesiredRunStatus(final String id, final String runStatus) { + final ConnectorDTO dto = new ConnectorDTO(); + dto.setId(id); + dto.setState(runStatus); + return dto; + } + + /** + * Applies an update to a connector. + * + * @param id The id of the connector to apply update to. + * @param requestConnectorEntity A connectorEntity containing the revision. + * @return A connectorEntity. + */ + @POST + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/apply-update") + @Operation( + summary = "Applies an update to a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "This will apply any pending configuration changes to the connector. The client can poll the connector endpoint to check when the update is complete.", + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response applyConnectorUpdate( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "The connector configuration with revision.", + required = true + ) final ConnectorEntity requestConnectorEntity) { + + if (requestConnectorEntity == null || requestConnectorEntity.getRevision() == null) { + throw new IllegalArgumentException("Connector entity with revision must be specified."); + } + + // ensure the ids are the same + if (requestConnectorEntity.getId() != null && !id.equals(requestConnectorEntity.getId())) { + throw new IllegalArgumentException(String.format("The connector id (%s) in the request body does not equal the " + + "connector id of the requested resource (%s).", requestConnectorEntity.getId(), id)); + } + + if (isReplicateRequest()) { + return replicate(HttpMethod.POST, requestConnectorEntity); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(requestConnectorEntity.isDisconnectedNodeAcknowledged()); + } + + // handle expects request (usually from the cluster manager) + final Revision requestRevision = getRevision(requestConnectorEntity, id); + return withWriteLock( + serviceFacade, + requestConnectorEntity, + requestRevision, + lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> { + // Verify the connector exists + serviceFacade.getConnector(id); + }, + (revision, connectorEntity) -> { + // apply the connector update + final ConnectorEntity entity = serviceFacade.applyConnectorUpdate(revision, id); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Discards the working configuration of a connector, reverting to the last applied configuration. + * + * @param version The revision is used to verify the client is working with the latest version of the flow. + * @param clientId Optional client id. If the client id is not specified, a new one will be generated. This value (whether specified or generated) is included in the response. + * @param id The id of the connector whose working configuration should be discarded. + * @return A connectorEntity. + */ + @DELETE + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/working-configuration") + @Operation( + summary = "Discards the working configuration of a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "This will discard any pending configuration changes for the connector and revert to the last applied configuration.", + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response discardConnectorUpdate( + @Parameter( + description = "The revision is used to verify the client is working with the latest version of the flow." + ) + @QueryParam(VERSION) final LongParameter version, + @Parameter( + description = "If the client id is not specified, new one will be generated. This value (whether specified or generated) is included in the response." + ) + @QueryParam(CLIENT_ID) @DefaultValue(StringUtils.EMPTY) final ClientIdParameter clientId, + @Parameter( + description = "Acknowledges that this node is disconnected to allow for mutable requests to proceed." + ) + @QueryParam(DISCONNECTED_NODE_ACKNOWLEDGED) @DefaultValue("false") final Boolean disconnectedNodeAcknowledged, + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.DELETE); + } else if (isDisconnectedFromCluster()) { + verifyDisconnectedNodeModification(disconnectedNodeAcknowledged); + } + + final ConnectorEntity requestConnectorEntity = new ConnectorEntity(); + requestConnectorEntity.setId(id); + + final Revision requestRevision = new Revision(version == null ? null : version.getLong(), clientId.getClientId(), id); + return withWriteLock( + serviceFacade, + requestConnectorEntity, + requestRevision, + lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> { + // Verify the connector exists + serviceFacade.getConnector(id); + }, + (revision, connectorEntity) -> { + final ConnectorEntity entity = serviceFacade.discardConnectorUpdate(revision, id); + populateRemainingConnectorEntityContent(entity); + + return generateOkResponse(entity).build(); + } + ); + } + + /** + * Retrieves the flow for a process group within the specified connector. + * + * @param connectorId The id of the connector + * @param processGroupId The process group id within the connector's hierarchy + * @param uiOnly Whether to return only UI-specific fields + * @return A processGroupFlowEntity + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{connectorId}/flow/process-groups/{processGroupId}") + @Operation( + summary = "Gets the flow for a process group within a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ProcessGroupFlowEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + }, + description = "Returns the flow for the specified process group within the connector's hierarchy. The processGroupId can be " + + "obtained from the managedProcessGroupId field of the ConnectorDTO for the root process group, or from child process " + + "groups within the flow. If the uiOnly query parameter is provided with a value of true, the returned entity may only " + + "contain fields that are necessary for rendering the NiFi User Interface. As such, the selected fields may change at " + + "any time, even during incremental releases, without warning. As a result, this parameter should not be provided by " + + "any client other than the UI." + ) + public Response getFlow( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("connectorId") final String connectorId, + @Parameter( + description = "The process group id.", + required = true + ) + @PathParam("processGroupId") final String processGroupId, + @Parameter( + description = "Whether to return only UI-specific fields" + ) + @QueryParam("uiOnly") @DefaultValue("false") final boolean uiOnly) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // authorize access to the connector + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the flow for the specified process group within the connector's hierarchy + final ProcessGroupFlowEntity entity = serviceFacade.getConnectorFlow(connectorId, processGroupId, uiOnly); + flowResource.populateRemainingFlowContent(entity.getProcessGroupFlow()); + return generateOkResponse(entity).build(); + } + + /** + * Retrieves all the controller services in the specified process group within a connector. + * + * @param connectorId The id of the connector + * @param processGroupId The process group id within the connector's hierarchy + * @param includeAncestorGroups Whether to include ancestor process groups + * @param includeDescendantGroups Whether to include descendant process groups + * @param includeReferences Whether to include services' referencing components in the response + * @return A controllerServicesEntity. + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{connectorId}/flow/process-groups/{processGroupId}/controller-services") + @Operation( + summary = "Gets all controller services for a process group within a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ControllerServicesEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + }, + description = "Returns the controller services for the specified process group within the connector's hierarchy. The processGroupId can be " + + "obtained from the managedProcessGroupId field of the ConnectorDTO for the root process group, or from child process " + + "groups within the flow." + ) + public Response getControllerServicesFromConnectorProcessGroup( + @Parameter(description = "The connector id.", required = true) + @PathParam("connectorId") final String connectorId, + @Parameter(description = "The process group id.", required = true) + @PathParam("processGroupId") final String processGroupId, + @Parameter(description = "Whether or not to include parent/ancestor process groups") + @QueryParam("includeAncestorGroups") + @DefaultValue("true") final boolean includeAncestorGroups, + @Parameter(description = "Whether or not to include descendant process groups") + @QueryParam("includeDescendantGroups") + @DefaultValue("false") final boolean includeDescendantGroups, + @Parameter(description = "Whether or not to include services' referencing components in the response") + @QueryParam("includeReferencingComponents") + @DefaultValue("true") final boolean includeReferences) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // authorize access to the connector + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the controller services for the specified process group within the connector's hierarchy + final Set controllerServices = serviceFacade.getConnectorControllerServices( + connectorId, processGroupId, includeAncestorGroups, includeDescendantGroups, includeReferences); + controllerServiceResource.populateRemainingControllerServiceEntitiesContent(controllerServices); + + // create the response entity + final ControllerServicesEntity entity = new ControllerServicesEntity(); + entity.setCurrentTime(new Date()); + entity.setControllerServices(controllerServices); + + // generate the response + return generateOkResponse(entity).build(); + } + + /** + * Retrieves the status for the process group managed by the specified connector. + * + * @param id The id of the connector + * @param recursive Optional recursive flag that defaults to false. If set to true, all descendant groups and the status of their content will be included. + * @param nodewise Whether to include breakdown per node + * @param clusterNodeId The id of a specific node to get status from + * @return A processGroupStatusEntity + * @throws InterruptedException if interrupted + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("/{id}/status") + @Operation( + summary = "Gets the status for the process group managed by a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ProcessGroupStatusEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + }, + description = "Returns the status for the process group managed by the specified connector. The status includes status for all descendent components. " + + "When invoked with recursive set to true, it will return the current status of every component in the connector's encapsulated flow." + ) + public Response getConnectorStatus( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String id, + @Parameter( + description = "Whether all descendant groups and the status of their content will be included. Optional, defaults to false" + ) + @QueryParam("recursive") @DefaultValue("false") final Boolean recursive, + @Parameter( + description = "Whether or not to include the breakdown per node. Optional, defaults to false" + ) + @QueryParam("nodewise") @DefaultValue("false") final Boolean nodewise, + @Parameter( + description = "The id of the node where to get the status." + ) + @QueryParam("clusterNodeId") final String clusterNodeId) throws InterruptedException { + + // ensure a valid request + if (Boolean.TRUE.equals(nodewise) && clusterNodeId != null) { + throw new IllegalArgumentException("Nodewise requests cannot be directed at a specific node."); + } + + if (isReplicateRequest()) { + // determine where this request should be sent + if (clusterNodeId == null) { + final NodeResponse nodeResponse = replicateNodeResponse(HttpMethod.GET); + final ProcessGroupStatusEntity entity = (ProcessGroupStatusEntity) nodeResponse.getUpdatedEntity(); + + // ensure there is an updated entity (result of merging) and prune the response as necessary + if (entity != null && !nodewise) { + entity.getProcessGroupStatus().setNodeSnapshots(null); + } + + return nodeResponse.getResponse(); + } else { + return replicate(HttpMethod.GET, clusterNodeId); + } + } + + // authorize access to the connector + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(id); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + // get the status for the connector's managed process group + final ProcessGroupStatusEntity entity = serviceFacade.getConnectorProcessGroupStatus(id, recursive); + return generateOkResponse(entity).build(); + } + + @POST + @Consumes(MediaType.APPLICATION_OCTET_STREAM) + @Produces(MediaType.APPLICATION_JSON) + @Path("{id}/assets") + @Operation( + summary = "Creates a new Asset in the given Connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = AssetEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "This endpoint will create a new Asset in the Connector. The Asset will be created with the given name and the contents of the file that is uploaded.", + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response createAsset( + @PathParam("id") final String connectorId, + @HeaderParam(FILENAME_HEADER) final String assetName, + @Parameter(description = "The contents of the asset", required = true) final InputStream assetContents) throws IOException { + + // Validate input + if (StringUtils.isBlank(assetName)) { + throw new IllegalArgumentException(FILENAME_HEADER + " header is required"); + } + if (assetContents == null) { + throw new IllegalArgumentException("Asset contents must be specified."); + } + + final String sanitizedAssetName = FileUtils.getSanitizedFilename(assetName); + if (!assetName.equals(sanitizedAssetName)) { + throw new IllegalArgumentException(FILENAME_HEADER + " header contains an invalid file name"); + } + + // If clustered and not all nodes are connected, do not allow creating an asset. + // Generally, we allow the flow to be modified when nodes are disconnected, but we do not allow creating an asset because + // the cluster has no mechanism for synchronizing those assets after the upload. + final ClusterCoordinator clusterCoordinator = getClusterCoordinator(); + if (clusterCoordinator != null) { + final Set disconnectedNodes = clusterCoordinator.getNodeIdentifiers(NodeConnectionState.CONNECTING, NodeConnectionState.DISCONNECTED, NodeConnectionState.DISCONNECTING); + if (!disconnectedNodes.isEmpty()) { + throw new IllegalStateException("Cannot create an Asset because the following %s nodes are not currently connected: %s".formatted(disconnectedNodes.size(), disconnectedNodes)); + } + } + + final NiFiUser currentUser = NiFiUserUtils.getNiFiUser(); + + // Verify Connector exists + serviceFacade.verifyCreateConnectorAsset(connectorId); + + // Authorize the request + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.WRITE, currentUser); + }); + + // If we need to replicate the request, we do so using the Upload Request Replicator, rather than the typical replicate() method. + // This is because Upload Request Replication works differently in that it needs to be able to replicate the InputStream multiple times, + // so it must create a file on disk to do so and then use the file's content to replicate the request. It also bypasses the two-phase + // commit process that is used for other requests because doing so would result in uploading the file twice to each node or providing a + // different request for each of the two phases. + + final long startTime = System.currentTimeMillis(); + final InputStream maxLengthInputStream = new MaxLengthInputStream(assetContents, MAX_ASSET_SIZE_BYTES); + + final AssetEntity assetEntity; + if (isReplicateRequest()) { + final String uploadRequestId = UUID.randomUUID().toString(); + final UploadRequest uploadRequest = new UploadRequest.Builder() + .user(NiFiUserUtils.getNiFiUser()) + .filename(sanitizedAssetName) + .identifier(uploadRequestId) + .contents(maxLengthInputStream) + .header(FILENAME_HEADER, sanitizedAssetName) + .header(CONTENT_TYPE_HEADER, UPLOAD_CONTENT_TYPE) + .header(RequestReplicationHeader.CLUSTER_ID_GENERATION_SEED.getHeader(), uploadRequestId) + .exampleRequestUri(getAbsolutePath()) + .responseClass(AssetEntity.class) + .successfulResponseStatus(HttpResponseStatus.OK.getCode()) + .build(); + assetEntity = uploadRequestReplicator.upload(uploadRequest); + } else { + final String assetId = generateUuid(); + logger.info("Creating asset [id={},name={}] in Connector [{}]", assetId, sanitizedAssetName, connectorId); + assetEntity = serviceFacade.createConnectorAsset(connectorId, assetId, sanitizedAssetName, maxLengthInputStream); + + final AssetDTO assetDTO = assetEntity.getAsset(); + final long elapsedTime = System.currentTimeMillis() - startTime; + logger.info("Creation of asset [id={},name={}] in Connector [{}] completed in {} ms", assetDTO.getId(), assetDTO.getName(), connectorId, elapsedTime); + } + + return generateOkResponse(assetEntity).build(); + } + + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("{id}/assets") + @Operation( + summary = "Lists the assets that belong to the Connector with the given ID", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = AssetsEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + description = "Lists the assets that belong to the Connector with the given ID.", + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response getAssets( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String connectorId + ) { + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + final List connectorAssets = serviceFacade.getConnectorAssets(connectorId); + logger.debug("Returning [{}] assets for connector [{}]", connectorAssets.size(), connectorId); + + final AssetsEntity assetsEntity = new AssetsEntity(); + assetsEntity.setAssets(connectorAssets); + + return generateOkResponse(assetsEntity).build(); + } + + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_OCTET_STREAM) + @Path("{id}/assets/{assetId}") + @Operation( + summary = "Retrieves the content of the asset with the given id for a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = byte[].class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /connectors/{uuid}") + } + ) + public Response getAssetContent( + @Parameter( + description = "The connector id.", + required = true + ) + @PathParam("id") final String connectorId, + @Parameter( + description = "The asset id.", + required = true + ) + @PathParam("assetId") final String assetId + ) { + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); + }); + + final Asset asset = serviceFacade.getConnectorAsset(assetId) + .orElseThrow(() -> new ResourceNotFoundException("Asset does not exist with id %s".formatted(assetId))); + + if (!asset.getOwnerIdentifier().equals(connectorId)) { + throw new ResourceNotFoundException("Asset does not exist with id %s".formatted(assetId)); + } + + if (!asset.getFile().exists()) { + throw new IllegalStateException("Content does not exist for asset with id %s".formatted(assetId)); + } + + final StreamingOutput streamingOutput = outputStream -> { + try (final InputStream assetInputStream = new FileInputStream(asset.getFile())) { + assetInputStream.transferTo(outputStream); + } + }; + + return generateOkResponse(streamingOutput) + .header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=\"%s\"", asset.getName())) + .build(); + } + + // ----------------- + // Processor State + // ----------------- + + /** + * Gets the state for a processor within a connector. + * + * @param connectorId the connector id + * @param processorId the processor id + * @return a ComponentStateEntity + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("{id}/processors/{processorId}/state") + @Operation( + summary = "Gets the state for a processor within a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ComponentStateEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response getConnectorProcessorState( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String connectorId, + @Parameter(description = "The processor id.", required = true) + @PathParam("processorId") final String processorId) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }); + + final ComponentStateDTO state = serviceFacade.getConnectorProcessorState(connectorId, processorId); + + final ComponentStateEntity entity = new ComponentStateEntity(); + entity.setComponentState(state); + + return generateOkResponse(entity).build(); + } + + /** + * Clears the state for a processor within a connector. + * + * @param connectorId the connector id + * @param processorId the processor id + * @return a ComponentStateEntity + */ + @POST + @Consumes({MediaType.APPLICATION_JSON, MediaType.WILDCARD}) + @Produces(MediaType.APPLICATION_JSON) + @Path("{id}/processors/{processorId}/state/clear-requests") + @Operation( + summary = "Clears the state for a processor within a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ComponentStateEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response clearConnectorProcessorState( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String connectorId, + @Parameter(description = "The processor id.", required = true) + @PathParam("processorId") final String processorId, + @Parameter(description = "Optional component state to perform a selective key removal. If omitted, clears all state.", required = false) + final ComponentStateEntity componentStateEntity) { + + if (isReplicateRequest()) { + if (componentStateEntity == null) { + return replicate(HttpMethod.POST); + } else { + return replicate(HttpMethod.POST, componentStateEntity); + } + } + + final ConnectorEntity requestConnectorEntity = new ConnectorEntity(); + requestConnectorEntity.setId(connectorId); + + return withWriteLock( + serviceFacade, + requestConnectorEntity, + lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> serviceFacade.verifyCanClearConnectorProcessorState(connectorId, processorId), + (connectorEntity) -> { + // clear state + final ComponentStateDTO expectedState = componentStateEntity == null ? null : componentStateEntity.getComponentState(); + final ComponentStateDTO state = serviceFacade.clearConnectorProcessorState(connectorEntity.getId(), processorId, expectedState); + + // generate the response entity + final ComponentStateEntity entity = new ComponentStateEntity(); + entity.setComponentState(state); + + // generate the response + return generateOkResponse(entity).build(); + } + ); + } + + // ----------------- + // Controller Service State + // ----------------- + + /** + * Gets the state for a controller service within a connector. + * + * @param connectorId the connector id + * @param controllerServiceId the controller service id + * @return a ComponentStateEntity + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("{id}/controller-services/{controllerServiceId}/state") + @Operation( + summary = "Gets the state for a controller service within a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ComponentStateEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response getConnectorControllerServiceState( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String connectorId, + @Parameter(description = "The controller service id.", required = true) + @PathParam("controllerServiceId") final String controllerServiceId) { + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + serviceFacade.authorizeAccess(lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }); + + final ComponentStateDTO state = serviceFacade.getConnectorControllerServiceState(connectorId, controllerServiceId); + + final ComponentStateEntity entity = new ComponentStateEntity(); + entity.setComponentState(state); + + return generateOkResponse(entity).build(); + } + + /** + * Clears the state for a controller service within a connector. + * + * @param connectorId the connector id + * @param controllerServiceId the controller service id + * @return a ComponentStateEntity + */ + @POST + @Consumes({MediaType.APPLICATION_JSON, MediaType.WILDCARD}) + @Produces(MediaType.APPLICATION_JSON) + @Path("{id}/controller-services/{controllerServiceId}/state/clear-requests") + @Operation( + summary = "Clears the state for a controller service within a connector", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ComponentStateEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The specified resource could not be found."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Write - /connectors/{uuid}") + } + ) + public Response clearConnectorControllerServiceState( + @Parameter(description = "The connector id.", required = true) + @PathParam("id") final String connectorId, + @Parameter(description = "The controller service id.", required = true) + @PathParam("controllerServiceId") final String controllerServiceId, + @Parameter(description = "Optional component state to perform a selective key removal. If omitted, clears all state.", required = false) + final ComponentStateEntity componentStateEntity) { + + if (isReplicateRequest()) { + if (componentStateEntity == null) { + return replicate(HttpMethod.POST); + } else { + return replicate(HttpMethod.POST, componentStateEntity); + } + } + + final ConnectorEntity requestConnectorEntity = new ConnectorEntity(); + requestConnectorEntity.setId(connectorId); + + return withWriteLock( + serviceFacade, + requestConnectorEntity, + lookup -> { + final Authorizable connector = lookup.getConnector(connectorId); + connector.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); + }, + () -> serviceFacade.verifyCanClearConnectorControllerServiceState(connectorId, controllerServiceId), + (connectorEntity) -> { + // clear state + final ComponentStateDTO expectedState = componentStateEntity == null ? null : componentStateEntity.getComponentState(); + final ComponentStateDTO state = serviceFacade.clearConnectorControllerServiceState(connectorEntity.getId(), controllerServiceId, expectedState); + + // generate the response entity + final ComponentStateEntity entity = new ComponentStateEntity(); + entity.setComponentState(state); + + // generate the response + return generateOkResponse(entity).build(); + } + ); + } + + // ----------------- + // setters + // ----------------- + + @Autowired + public void setServiceFacade(final NiFiServiceFacade serviceFacade) { + this.serviceFacade = serviceFacade; + } + + @Autowired + public void setAuthorizer(final Authorizer authorizer) { + this.authorizer = authorizer; + } + + @Autowired + public void setFlowResource(final FlowResource flowResource) { + this.flowResource = flowResource; + } + + @Autowired + public void setControllerServiceResource(final ControllerServiceResource controllerServiceResource) { + this.controllerServiceResource = controllerServiceResource; + } + + @Autowired(required = false) + public void setUploadRequestReplicator(final UploadRequestReplicator uploadRequestReplicator) { + this.uploadRequestReplicator = uploadRequestReplicator; + } + +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowFileQueueResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowFileQueueResource.java index c018b88a464d..399f6f064455 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowFileQueueResource.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowFileQueueResource.java @@ -307,7 +307,7 @@ public Response createFlowFileListing( serviceFacade, requestConnectionEntity, lookup -> { - final ConnectionAuthorizable connAuth = lookup.getConnection(id); + final ConnectionAuthorizable connAuth = lookup.getConnection(id, true); final Authorizable dataAuthorizable = connAuth.getSourceData(); dataAuthorizable.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); }, @@ -374,7 +374,7 @@ public Response getListingRequest( // authorize access serviceFacade.authorizeAccess(lookup -> { - final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId); + final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId, true); final Authorizable dataAuthorizable = connAuth.getSourceData(); dataAuthorizable.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); }); @@ -435,7 +435,7 @@ public Response deleteListingRequest( serviceFacade, new ListingEntity(connectionId, listingRequestId), lookup -> { - final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId); + final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId, true); final Authorizable dataAuthorizable = connAuth.getSourceData(); dataAuthorizable.authorize(authorizer, RequestAction.READ, NiFiUserUtils.getNiFiUser()); }, @@ -522,7 +522,7 @@ public Response createDropRequest( serviceFacade, requestConnectionEntity, lookup -> { - final ConnectionAuthorizable connAuth = lookup.getConnection(id); + final ConnectionAuthorizable connAuth = lookup.getConnection(id, true); final Authorizable dataAuthorizable = connAuth.getSourceData(); dataAuthorizable.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); }, @@ -589,7 +589,7 @@ public Response getDropRequest( // authorize access serviceFacade.authorizeAccess(lookup -> { - final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId); + final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId, true); final Authorizable dataAuthorizable = connAuth.getSourceData(); dataAuthorizable.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); }); @@ -650,7 +650,7 @@ public Response removeDropRequest( serviceFacade, new DropEntity(connectionId, dropRequestId), lookup -> { - final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId); + final ConnectionAuthorizable connAuth = lookup.getConnection(connectionId, true); final Authorizable dataAuthorizable = connAuth.getSourceData(); dataAuthorizable.authorize(authorizer, RequestAction.WRITE, NiFiUserUtils.getNiFiUser()); }, diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowResource.java index d94f8b05a4e8..959ef1e8ebd2 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowResource.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/FlowResource.java @@ -54,6 +54,7 @@ import org.apache.nifi.bundle.Bundle; import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.bundle.BundleDetails; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.FlowAnalysisRuleDefinition; import org.apache.nifi.c2.protocol.component.api.FlowRegistryClientDefinition; @@ -115,6 +116,9 @@ import org.apache.nifi.web.api.entity.ComponentHistoryEntity; import org.apache.nifi.web.api.entity.ConnectionStatisticsEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorTypesEntity; +import org.apache.nifi.web.api.entity.ConnectorsEntity; import org.apache.nifi.web.api.entity.ContentViewerEntity; import org.apache.nifi.web.api.entity.ControllerBulletinsEntity; import org.apache.nifi.web.api.entity.ControllerServiceEntity; @@ -155,6 +159,7 @@ import org.apache.nifi.web.api.entity.ScheduleComponentsEntity; import org.apache.nifi.web.api.entity.SearchResultsEntity; import org.apache.nifi.web.api.entity.StatusHistoryEntity; +import org.apache.nifi.web.api.entity.StepDocumentationEntity; import org.apache.nifi.web.api.entity.VersionedFlowEntity; import org.apache.nifi.web.api.entity.VersionedFlowSnapshotMetadataEntity; import org.apache.nifi.web.api.entity.VersionedFlowSnapshotMetadataSetEntity; @@ -231,6 +236,7 @@ public class FlowResource extends ApplicationResource { private ControllerServiceResource controllerServiceResource; private ReportingTaskResource reportingTaskResource; private ParameterProviderResource parameterProviderResource; + private ConnectorResource connectorResource; @Context private ServletContext servletContext; @@ -240,7 +246,7 @@ public class FlowResource extends ApplicationResource { * * @param flow group */ - private void populateRemainingFlowContent(ProcessGroupFlowDTO flow) { + public void populateRemainingFlowContent(final ProcessGroupFlowDTO flow) { FlowDTO flowStructure = flow.getFlow(); // populate the remaining fields for the processors, connections, process group refs, remote process groups, and labels if appropriate @@ -776,6 +782,105 @@ public Response getParameterProviders() { return generateOkResponse(entity).build(); } + // ---------- + // connectors + // ---------- + + /** + * Retrieves all the of connectors in this NiFi. + * + * @return A connectorsEntity. + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("connectors") + @Operation( + summary = "Gets all connectors", + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorsEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /flow") + } + ) + public Response getConnectors() { + + authorizeFlow(); + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // get all the connectors + final Set connectors = serviceFacade.getConnectors(); + connectorResource.populateRemainingConnectorEntitiesContent(connectors); + + // create the response entity + final ConnectorsEntity entity = new ConnectorsEntity(); + entity.setCurrentTime(new Date()); + entity.setConnectors(connectors); + + // generate the response + return generateOkResponse(entity).build(); + } + + /** + * Retrieves the types of connectors that this NiFi supports. + * + * @return A connectorTypesEntity. + * @throws InterruptedException if interrupted + */ + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("connector-types") + @Operation( + summary = "Retrieves the types of connectors that this NiFi supports", + description = NON_GUARANTEED_ENDPOINT, + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorTypesEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "409", description = "The request was valid but NiFi was not in the appropriate state to process it.") + }, + security = { + @SecurityRequirement(name = "Read - /flow") + } + ) + public Response getConnectorTypes( + @Parameter( + description = "If specified, will only return types that are a member of this bundle group." + ) + @QueryParam("bundleGroupFilter") String bundleGroupFilter, + @Parameter( + description = "If specified, will only return types that are a member of this bundle artifact." + ) + @QueryParam("bundleArtifactFilter") String bundleArtifactFilter, + @Parameter( + description = "If specified, will only return types whose fully qualified classname matches." + ) + @QueryParam("type") String typeFilter) throws InterruptedException { + + authorizeFlow(); + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // create response entity + final ConnectorTypesEntity entity = new ConnectorTypesEntity(); + entity.setConnectorTypes(serviceFacade.getConnectorTypes(bundleGroupFilter, bundleArtifactFilter, typeFilter)); + + // generate the response + return generateOkResponse(entity).build(); + } + // --------------- // reporting-tasks // --------------- @@ -2180,6 +2285,60 @@ public Response getFlowAnalysisRuleDefinition( return generateOkResponse(entity).build(); } + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("connector-definition/{group}/{artifact}/{version}/{type}") + @Operation( + summary = "Retrieves the Connector Definition for the specified component type.", + description = NON_GUARANTEED_ENDPOINT, + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = ConnectorDefinition.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The connector definition for the coordinates could not be located.") + }, + security = { + @SecurityRequirement(name = "Read - /flow") + } + ) + public Response getConnectorDefinition( + @Parameter( + description = "The bundle group", + required = true + ) + @PathParam("group") final String group, + @Parameter( + description = "The bundle artifact", + required = true + ) + @PathParam("artifact") final String artifact, + @Parameter( + description = "The bundle version", + required = true + ) + @PathParam("version") final String version, + @Parameter( + description = "The connector type", + required = true + ) + @PathParam("type") final String type + ) throws InterruptedException { + + authorizeFlow(); + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + // create response entity + final ConnectorDefinition entity = serviceFacade.getConnectorDefinition(group, artifact, version, type); + + // generate the response + return generateOkResponse(entity).build(); + } + @GET @Consumes(MediaType.WILDCARD) @Produces(MediaType.APPLICATION_JSON) @@ -2236,6 +2395,50 @@ public Response getAdditionalDetails( return generateOkResponse(entity).build(); } + @GET + @Consumes(MediaType.WILDCARD) + @Produces(MediaType.APPLICATION_JSON) + @Path("steps/{group}/{artifact}/{version}/{connectorType}/{stepName}") + @Operation( + summary = "Retrieves the step documentation for the specified Connector configuration step.", + description = NON_GUARANTEED_ENDPOINT, + responses = { + @ApiResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = StepDocumentationEntity.class))), + @ApiResponse(responseCode = "400", description = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), + @ApiResponse(responseCode = "401", description = "Client could not be authenticated."), + @ApiResponse(responseCode = "403", description = "Client is not authorized to make this request."), + @ApiResponse(responseCode = "404", description = "The step documentation for the coordinates could not be located.") + }, + security = { + @SecurityRequirement(name = "Read - /flow") + } + ) + public Response getStepDocumentation( + @Parameter(description = "The bundle group", required = true) + @PathParam("group") final String group, + @Parameter(description = "The bundle artifact", required = true) + @PathParam("artifact") final String artifact, + @Parameter(description = "The bundle version", required = true) + @PathParam("version") final String version, + @Parameter(description = "The fully qualified Connector type", required = true) + @PathParam("connectorType") final String connectorType, + @Parameter(description = "The configuration step name", required = true) + @PathParam("stepName") final String stepName + ) throws InterruptedException { + + authorizeFlow(); + + if (isReplicateRequest()) { + return replicate(HttpMethod.GET); + } + + final String stepDocumentation = serviceFacade.getStepDocumentation(group, artifact, version, connectorType, stepName); + final StepDocumentationEntity entity = new StepDocumentationEntity(); + entity.setStepDocumentation(stepDocumentation); + + return generateOkResponse(entity).build(); + } + /** * Retrieves the types of parameter providers that this NiFi supports. * @@ -4151,6 +4354,11 @@ public void setParameterProviderResource(final ParameterProviderResource paramet this.parameterProviderResource = parameterProviderResource; } + @Autowired + public void setConnectorResource(final ConnectorResource connectorResource) { + this.connectorResource = connectorResource; + } + @Autowired public void setAuthorizer(Authorizer authorizer) { this.authorizer = authorizer; diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ParameterContextResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ParameterContextResource.java index 51ece06c1f5f..522fd359741a 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ParameterContextResource.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ParameterContextResource.java @@ -537,7 +537,7 @@ public Response getAssetContent( final Asset asset = assetManager.getAsset(assetId) .orElseThrow(() -> new ResourceNotFoundException("Asset does not exist with the id %s".formatted(assetId))); - if (!asset.getParameterContextIdentifier().equals(parameterContextId)) { + if (!asset.getOwnerIdentifier().equals(parameterContextId)) { throw new ResourceNotFoundException("Asset does not exist with id %s".formatted(assetId)); } @@ -811,7 +811,7 @@ private void validateAssetReferences(final ParameterContextDTO parameterContextD } final Asset asset = assetManager.getAsset(referencedAsset.getId()) .orElseThrow(() -> new IllegalArgumentException("Request contains a reference to an Asset (%s) that does not exist".formatted(referencedAsset))); - if (!asset.getParameterContextIdentifier().equals(parameterContextDto.getId())) { + if (!asset.getOwnerIdentifier().equals(parameterContextDto.getId())) { throw new IllegalArgumentException("Request contains a reference to an Asset (%s) that does not exist in Parameter Context (%s)" .formatted(referencedAsset, parameterContextDto.getId())); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ProcessGroupResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ProcessGroupResource.java index 6d55f9d0651a..435d6846956d 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ProcessGroupResource.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ProcessGroupResource.java @@ -870,7 +870,7 @@ public Response removeDropRequest( } private void authorizeHandleDropAllFlowFilesRequest(String processGroupId, AuthorizableLookup lookup) { - final ProcessGroupAuthorizable processGroup = lookup.getProcessGroup(processGroupId); + final ProcessGroupAuthorizable processGroup = lookup.getProcessGroup(processGroupId, true); authorizeProcessGroup(processGroup, authorizer, lookup, RequestAction.READ, false, false, false, false, false); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java index 6954dc6607a9..945847b6398d 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java @@ -62,9 +62,27 @@ import org.apache.nifi.cluster.manager.StatusMerger; import org.apache.nifi.cluster.protocol.NodeIdentifier; import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; import org.apache.nifi.components.PropertyDependency; import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConfigurationStepDependency; +import org.apache.nifi.components.connector.ConnectorAction; +import org.apache.nifi.components.connector.ConnectorConfiguration; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.NamedStepConfiguration; +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.components.connector.StringLiteralValue; import org.apache.nifi.components.state.Scope; import org.apache.nifi.components.state.StateMap; import org.apache.nifi.components.validation.ValidationState; @@ -129,6 +147,7 @@ import org.apache.nifi.groups.RemoteProcessGroup; import org.apache.nifi.groups.RemoteProcessGroupCounts; import org.apache.nifi.history.History; +import org.apache.nifi.manifest.RuntimeManifestService; import org.apache.nifi.nar.ExtensionDefinition; import org.apache.nifi.nar.ExtensionManager; import org.apache.nifi.nar.NarClassLoadersHolder; @@ -221,6 +240,8 @@ import org.apache.nifi.web.api.dto.status.ConnectionStatusDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusPredictionsSnapshotDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusSnapshotDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.PortStatusDTO; import org.apache.nifi.web.api.dto.status.PortStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.ProcessGroupStatusDTO; @@ -263,6 +284,7 @@ import java.nio.charset.StandardCharsets; import java.text.Collator; import java.text.NumberFormat; +import java.time.Duration; import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; @@ -302,6 +324,8 @@ public final class DtoFactory { private EntityFactory entityFactory; private Authorizer authorizer; private ExtensionManager extensionManager; + private ConnectorRepository connectorRepository; + private RuntimeManifestService runtimeManifestService; public ControllerConfigurationDTO createControllerConfigurationDto(final ControllerFacade controllerFacade) { final ControllerConfigurationDTO dto = new ControllerConfigurationDTO(); @@ -5200,14 +5224,22 @@ public void setEntityFactory(final EntityFactory entityFactory) { this.entityFactory = entityFactory; } - public void setBulletinRepository(BulletinRepository bulletinRepository) { + public void setBulletinRepository(final BulletinRepository bulletinRepository) { this.bulletinRepository = bulletinRepository; } - public void setExtensionManager(ExtensionManager extensionManager) { + public void setExtensionManager(final ExtensionManager extensionManager) { this.extensionManager = extensionManager; } + public void setConnectorRepository(final ConnectorRepository connectorRepository) { + this.connectorRepository = connectorRepository; + } + + public void setRuntimeManifestService(RuntimeManifestService runtimeManifestService) { + this.runtimeManifestService = runtimeManifestService; + } + private ProcessingPerformanceStatusDTO createProcessingPerformanceStatusDTO(final ProcessingPerformanceStatus performanceStatus) { final ProcessingPerformanceStatusDTO performanceStatusDTO = new ProcessingPerformanceStatusDTO(); @@ -5227,4 +5259,352 @@ private String getDigest(final String name) { final byte[] digest = MessageDigestUtils.getDigest(bytes); return HexFormat.of().formatHex(digest); } + + public ConnectorDTO createConnectorDto(final ConnectorNode connector) { + if (connector == null) { + return null; + } + + final ConnectorDTO dto = new ConnectorDTO(); + dto.setId(connector.getIdentifier()); + dto.setName(connector.getName()); + + final ValidationState validationState = connector.getValidationState(); + dto.setValidationStatus(validationState.getStatus().name()); + dto.setValidationErrors(convertValidationErrors(validationState.getValidationErrors())); + dto.setType(connector.getCanonicalClassName()); + dto.setExtensionMissing(connector.isExtensionMissing()); + + dto.setBundle(createBundleDto(connector.getBundleCoordinate())); + dto.setState(connector.getCurrentState().name()); + + final FrameworkFlowContext activeFlowContext = connector.getActiveFlowContext(); + dto.setManagedProcessGroupId(activeFlowContext.getManagedProcessGroup().getIdentifier()); + dto.setActiveConfiguration(createConnectorConfigurationDtoFromFlowContext(connector, activeFlowContext)); + dto.setWorkingConfiguration(createConnectorConfigurationDtoFromFlowContext(connector, connector.getWorkingFlowContext())); + dto.setAvailableActions(createConnectorActionDtos(connector)); + + return dto; + } + + public ConnectorStatusDTO createConnectorStatusDto(final ConnectorNode connectorNode, final ProcessGroupStatus managedGroupStatus) { + if (connectorNode == null) { + return null; + } + + final ConnectorStatusDTO statusDto = new ConnectorStatusDTO(); + statusDto.setId(connectorNode.getIdentifier()); + statusDto.setName(connectorNode.getName()); + statusDto.setType(connectorNode.getCanonicalClassName()); + statusDto.setRunStatus(connectorNode.getCurrentState().name()); + statusDto.setValidationStatus(connectorNode.getValidationStatus().name()); + statusDto.setStatsLastRefreshed(new Date()); + + final ConnectorStatusSnapshotDTO snapshot = new ConnectorStatusSnapshotDTO(); + statusDto.setAggregateSnapshot(snapshot); + + snapshot.setId(connectorNode.getIdentifier()); + snapshot.setName(connectorNode.getName()); + snapshot.setType(connectorNode.getCanonicalClassName()); + snapshot.setRunStatus(connectorNode.getCurrentState().name()); + + // Populate all status metrics from the managed process group + if (managedGroupStatus != null) { + snapshot.setFlowFilesSent(managedGroupStatus.getFlowFilesSent()); + snapshot.setBytesSent(managedGroupStatus.getBytesSent()); + snapshot.setFlowFilesReceived(managedGroupStatus.getFlowFilesReceived()); + snapshot.setBytesReceived(managedGroupStatus.getBytesReceived()); + snapshot.setBytesRead(managedGroupStatus.getBytesRead()); + snapshot.setBytesWritten(managedGroupStatus.getBytesWritten()); + snapshot.setFlowFilesQueued(managedGroupStatus.getQueuedCount()); + snapshot.setBytesQueued(managedGroupStatus.getQueuedContentSize()); + snapshot.setActiveThreadCount(managedGroupStatus.getActiveThreadCount()); + + final ProcessingPerformanceStatus performanceStatus = managedGroupStatus.getProcessingPerformanceStatus(); + if (performanceStatus != null) { + snapshot.setProcessingPerformanceStatus(createProcessingPerformanceStatusDTO(performanceStatus)); + } + } + + // Populate idle status + final Optional idleDuration = connectorNode.getIdleDuration(); + if (idleDuration.isPresent()) { + snapshot.setIdle(true); + final long idleMillis = idleDuration.get().toMillis(); + snapshot.setIdleDurationMillis(idleMillis); + snapshot.setIdleDuration(FormatUtils.formatHoursMinutesSeconds(idleMillis, TimeUnit.MILLISECONDS)); + } else { + snapshot.setIdle(false); + } + + StatusMerger.updatePrettyPrintedFields(snapshot); + + return statusDto; + } + + private List createConnectorActionDtos(final ConnectorNode connector) { + return connector.getAvailableActions().stream() + .map(this::createConnectorActionDto) + .collect(Collectors.toList()); + } + + private ConnectorActionDTO createConnectorActionDto(final ConnectorAction action) { + final ConnectorActionDTO dto = new ConnectorActionDTO(); + dto.setName(action.getName()); + dto.setDescription(action.getDescription()); + dto.setAllowed(action.isAllowed()); + dto.setReasonNotAllowed(action.getReasonNotAllowed()); + return dto; + } + + private ConnectorConfigurationDTO createConnectorConfigurationDtoFromFlowContext(final ConnectorNode connector, final FrameworkFlowContext flowContext) { + final List configurationSteps = connector.getConfigurationSteps(); + if (configurationSteps == null || configurationSteps.isEmpty()) { + return null; + } + + if (flowContext == null || flowContext.getConfigurationContext() == null) { + return null; + } + + final BundleCoordinate bundleCoordinate = connector.getBundleCoordinate(); + final String connectorType = connector.getCanonicalClassName(); + + final Set stepsWithDocumentation = discoverStepsWithDocumentation(bundleCoordinate, connectorType); + + final ConnectorConfiguration configuration = flowContext.getConfigurationContext().toConnectorConfiguration(); + final ConnectorConfigurationDTO dto = new ConnectorConfigurationDTO(); + final List configurationStepDtos = configurationSteps.stream() + .map(step -> createConfigurationStepConfigurationDtoFromStep(step, configuration, stepsWithDocumentation)) + .collect(Collectors.toList()); + dto.setConfigurationStepConfigurations(configurationStepDtos); + return dto; + } + + private Set discoverStepsWithDocumentation(final BundleCoordinate bundleCoordinate, final String connectorType) { + if (runtimeManifestService == null) { + return Collections.emptySet(); + } + + try { + final Map stepDocs = runtimeManifestService.discoverStepDocumentation( + bundleCoordinate.getGroup(), bundleCoordinate.getId(), bundleCoordinate.getVersion(), connectorType); + return stepDocs.keySet(); + } catch (final Exception e) { + logger.debug("Unable to discover step documentation for connector [{}]: {}", connectorType, e.getMessage()); + return Collections.emptySet(); + } + } + + private ConfigurationStepConfigurationDTO createConfigurationStepConfigurationDtoFromStep(final ConfigurationStep step, + final ConnectorConfiguration configuration, + final Set stepsWithDocumentation) { + if (step == null) { + return null; + } + + final ConfigurationStepConfigurationDTO dto = new ConfigurationStepConfigurationDTO(); + dto.setConfigurationStepName(step.getName()); + dto.setConfigurationStepDescription(step.getDescription()); + dto.setDocumented(stepsWithDocumentation.contains(step.getName())); + + // Get the current configuration values for this step from the flat StepConfiguration + final StepConfiguration stepConfig = configuration.getNamedStepConfigurations().stream() + .filter(c -> step.getName().equals(c.stepName())) + .map(NamedStepConfiguration::configuration) + .findFirst() + .orElse(null); + + // Convert property groups from the schema, merging in current values from flat stepConfig + final List propertyGroupDtos = step.getPropertyGroups().stream() + .map(propertyGroup -> createPropertyGroupConfigurationDtoFromGroup(propertyGroup, stepConfig)) + .collect(Collectors.toList()); + dto.setPropertyGroupConfigurations(propertyGroupDtos); + + // Convert step dependencies + final Set dependencyDtos = step.getDependencies().stream() + .map(this::createConfigurationStepDependencyDto) + .collect(Collectors.toSet()); + dto.setDependencies(dependencyDtos); + + return dto; + } + + private ConfigurationStepDependencyDTO createConfigurationStepDependencyDto(final ConfigurationStepDependency dependency) { + final ConfigurationStepDependencyDTO dto = new ConfigurationStepDependencyDTO(); + dto.setStepName(dependency.getStepName()); + dto.setPropertyName(dependency.getPropertyName()); + dto.setDependentValues(dependency.getDependentValues()); + return dto; + } + + private PropertyGroupConfigurationDTO createPropertyGroupConfigurationDtoFromGroup(final ConnectorPropertyGroup propertyGroup, final StepConfiguration stepConfig) { + if (propertyGroup == null) { + return null; + } + + final PropertyGroupConfigurationDTO dto = new PropertyGroupConfigurationDTO(); + dto.setPropertyGroupName(propertyGroup.getName()); + dto.setPropertyGroupDescription(propertyGroup.getDescription()); + + // Convert property descriptors from the schema, keyed by property name + // Use LinkedHashMap to preserve the order from the connector + final Map propertyDescriptorMap = new LinkedHashMap<>(); + for (final ConnectorPropertyDescriptor propertyDescriptor : propertyGroup.getProperties()) { + final ConnectorPropertyDescriptorDTO descriptorDto = createConnectorPropertyDescriptorDto(propertyDescriptor); + propertyDescriptorMap.put(descriptorDto.getName(), descriptorDto); + } + dto.setPropertyDescriptors(propertyDescriptorMap); + + // Get the current property values for this group's properties from flat stepConfig + // Use LinkedHashMap to preserve the order from the connector + final Map propertyValues = new LinkedHashMap<>(); + if (stepConfig != null) { + // For each property in this group, check if there's a value in the flat stepConfig + for (final ConnectorPropertyDescriptor propertyDescriptor : propertyGroup.getProperties()) { + final ConnectorValueReference valueRef = stepConfig.getPropertyValue(propertyDescriptor.getName()); + if (valueRef != null) { + propertyValues.put(propertyDescriptor.getName(), createConnectorValueReferenceDto(valueRef)); + } + } + } + + dto.setPropertyValues(propertyValues); + return dto; + } + + private ConnectorValueReferenceDTO createConnectorValueReferenceDto(final ConnectorValueReference valueReference) { + if (valueReference == null) { + return null; + } + + final ConnectorValueReferenceDTO dto = new ConnectorValueReferenceDTO(); + dto.setValueType(valueReference.getValueType() != null ? valueReference.getValueType().name() : null); + + switch (valueReference) { + case StringLiteralValue stringLiteral -> dto.setValue(stringLiteral.getValue()); + case AssetReference assetRef -> { + final List assetReferences = assetRef.getAssetIdentifiers().stream() + .map(this::createConnectorAssetReferenceDto) + .toList(); + dto.setAssetReferences(assetReferences); + } + case SecretReference secretRef -> { + dto.setSecretProviderId(secretRef.getProviderId()); + dto.setSecretProviderName(secretRef.getProviderName()); + dto.setSecretName(secretRef.getSecretName()); + dto.setFullyQualifiedSecretName(secretRef.getFullyQualifiedName()); + } + } + + return dto; + } + + private AssetReferenceDTO createConnectorAssetReferenceDto(final String assetId) { + final String assetName = connectorRepository.getAsset(assetId).map(Asset::getName).orElse(assetId); + return new AssetReferenceDTO(assetId, assetName); + } + + private ConnectorPropertyDescriptorDTO createConnectorPropertyDescriptorDto(final ConnectorPropertyDescriptor propertyDescriptor) { + if (propertyDescriptor == null) { + return null; + } + + final ConnectorPropertyDescriptorDTO dto = new ConnectorPropertyDescriptorDTO(); + dto.setName(propertyDescriptor.getName()); + dto.setDescription(propertyDescriptor.getDescription()); + dto.setDefaultValue(propertyDescriptor.getDefaultValue()); + dto.setRequired(propertyDescriptor.isRequired()); + dto.setType(propertyDescriptor.getType() != null ? propertyDescriptor.getType().name() : null); + dto.setAllowableValuesFetchable(propertyDescriptor.isAllowableValuesFetchable()); + + // Convert allowable values if present + if (propertyDescriptor.getAllowableValues() != null && !propertyDescriptor.getAllowableValues().isEmpty()) { + final List allowableValueEntities = propertyDescriptor.getAllowableValues().stream() + .map(describedValue -> { + final AllowableValueDTO allowableValueDto = new AllowableValueDTO(); + allowableValueDto.setValue(describedValue.getValue()); + allowableValueDto.setDisplayName(describedValue.getDisplayName()); + allowableValueDto.setDescription(describedValue.getDescription()); + + final AllowableValueEntity entity = new AllowableValueEntity(); + entity.setAllowableValue(allowableValueDto); + entity.setCanRead(true); + return entity; + }) + .collect(Collectors.toList()); + dto.setAllowableValues(allowableValueEntities); + } + + // Convert dependencies if present + if (propertyDescriptor.getDependencies() != null && !propertyDescriptor.getDependencies().isEmpty()) { + final Set dependencyDtos = propertyDescriptor.getDependencies().stream() + .map(dependency -> { + final ConnectorPropertyDependencyDTO dependencyDto = new ConnectorPropertyDependencyDTO(); + dependencyDto.setPropertyName(dependency.getPropertyName()); + dependencyDto.setDependentValues(dependency.getDependentValues()); + return dependencyDto; + }) + .collect(Collectors.toSet()); + dto.setDependencies(dependencyDtos); + } + + return dto; + } + + /** + * Creates an AllowableValueDTO from the specified AllowableValue. + * + * @param allowableValue the allowable value + * @return the DTO + */ + public AllowableValueDTO createAllowableValueDto(final DescribedValue allowableValue) { + if (allowableValue == null) { + return null; + } + + final AllowableValueDTO dto = new AllowableValueDTO(); + dto.setValue(allowableValue.getValue()); + dto.setDisplayName(allowableValue.getDisplayName()); + dto.setDescription(allowableValue.getDescription()); + return dto; + } + + /** + * Creates a ConfigVerificationResultDTO from the specified ConfigVerificationResult. + * + * @param result the verification result + * @return the DTO + */ + public ConfigVerificationResultDTO createConfigVerificationResultDto(final ConfigVerificationResult result) { + final ConfigVerificationResultDTO dto = new ConfigVerificationResultDTO(); + dto.setExplanation(result.getExplanation()); + dto.setOutcome(result.getOutcome().name()); + dto.setVerificationStepName(result.getVerificationStepName()); + dto.setSubject(result.getSubject()); + return dto; + } + + /** + * Creates a SecretDTO from the specified Secret. Note: The secret value is intentionally not included + * in the DTO for security reasons. + * + * @param secret the secret + * @return the DTO containing only the secret's metadata + */ + public SecretDTO createSecretDto(final Secret secret) { + if (secret == null) { + return null; + } + + final SecretDTO dto = new SecretDTO(); + dto.setProviderId(secret.getProviderId()); + dto.setProviderName(secret.getProviderName()); + dto.setGroupName(secret.getGroupName()); + dto.setName(secret.getName()); + dto.setFullyQualifiedName(secret.getFullyQualifiedName()); + dto.setDescription(secret.getDescription()); + return dto; + } + } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java index f17f2781440d..be52686de488 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java @@ -24,6 +24,7 @@ import org.apache.nifi.web.api.dto.status.ConnectionStatisticsSnapshotDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusDTO; import org.apache.nifi.web.api.dto.status.ConnectionStatusSnapshotDTO; +import org.apache.nifi.web.api.dto.status.ConnectorStatusDTO; import org.apache.nifi.web.api.dto.status.ControllerServiceStatusDTO; import org.apache.nifi.web.api.dto.status.FlowAnalysisRuleStatusDTO; import org.apache.nifi.web.api.dto.status.PortStatusDTO; @@ -44,11 +45,15 @@ import org.apache.nifi.web.api.entity.BulletinEntity; import org.apache.nifi.web.api.entity.ComponentReferenceEntity; import org.apache.nifi.web.api.entity.ComponentValidationResultEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; import org.apache.nifi.web.api.entity.ConnectionEntity; import org.apache.nifi.web.api.entity.ConnectionStatisticsEntity; import org.apache.nifi.web.api.entity.ConnectionStatisticsSnapshotEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; import org.apache.nifi.web.api.entity.ConnectionStatusSnapshotEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; import org.apache.nifi.web.api.entity.ControllerConfigurationEntity; import org.apache.nifi.web.api.entity.ControllerServiceEntity; import org.apache.nifi.web.api.entity.ControllerServiceReferencingComponentEntity; @@ -80,6 +85,7 @@ import org.apache.nifi.web.api.entity.RemoteProcessGroupStatusEntity; import org.apache.nifi.web.api.entity.RemoteProcessGroupStatusSnapshotEntity; import org.apache.nifi.web.api.entity.ReportingTaskEntity; +import org.apache.nifi.web.api.entity.SecretsEntity; import org.apache.nifi.web.api.entity.SnippetEntity; import org.apache.nifi.web.api.entity.StatusHistoryEntity; import org.apache.nifi.web.api.entity.TenantEntity; @@ -124,6 +130,25 @@ public ProcessorStatusEntity createProcessorStatusEntity(final ProcessorStatusDT return entity; } + public ConnectorEntity createConnectorEntity(final ConnectorDTO dto, + final RevisionDTO revision, + final PermissionsDTO permissions, + final PermissionsDTO operatePermissions, + final ConnectorStatusDTO status) { + final ConnectorEntity entity = new ConnectorEntity(); + entity.setRevision(revision); + if (dto != null) { + entity.setPermissions(permissions); + entity.setOperatePermissions(operatePermissions); + entity.setStatus(status); + entity.setId(dto.getId()); + if (permissions != null && permissions.getCanRead()) { + entity.setComponent(dto); + } + } + return entity; + } + public ProcessorStatusSnapshotEntity createProcessorStatusSnapshotEntity(final ProcessorStatusSnapshotDTO status, final PermissionsDTO permissions) { final ProcessorStatusSnapshotEntity entity = new ProcessorStatusSnapshotEntity(); entity.setId(status.getId()); @@ -842,4 +867,68 @@ public FlowRegistryBranchEntity createBranchEntity(final FlowRegistryBranchDTO d public NarSummaryEntity createNarSummaryEntity(final NarSummaryDTO narSummaryDTO) { return new NarSummaryEntity(narSummaryDTO); } + + public ConfigurationStepNamesEntity createConfigurationStepNamesEntity(final ConnectorDTO connectorDto) { + final ConfigurationStepNamesEntity entity = new ConfigurationStepNamesEntity(); + + final List configurationStepNames; + if (connectorDto != null + && connectorDto.getWorkingConfiguration() != null + && connectorDto.getWorkingConfiguration().getConfigurationStepConfigurations() != null) { + + configurationStepNames = connectorDto.getWorkingConfiguration().getConfigurationStepConfigurations() + .stream() + .map(ConfigurationStepConfigurationDTO::getConfigurationStepName) + .collect(Collectors.toList()); + } else { + configurationStepNames = List.of(); + } + + entity.setConfigurationStepNames(configurationStepNames); + return entity; + } + + public ConfigurationStepEntity createConfigurationStepEntity(final ConnectorDTO connectorDto, + final String configurationStepName, final RevisionDTO parentConnectorRevision) { + ConfigurationStepConfigurationDTO foundConfigurationStep = null; + + if (connectorDto != null + && connectorDto.getWorkingConfiguration() != null + && connectorDto.getWorkingConfiguration().getConfigurationStepConfigurations() != null) { + + foundConfigurationStep = connectorDto.getWorkingConfiguration().getConfigurationStepConfigurations() + .stream() + .filter(step -> configurationStepName.equals(step.getConfigurationStepName())) + .findFirst() + .orElse(null); + } + + if (foundConfigurationStep == null) { + final String connectorId = connectorDto != null ? connectorDto.getId() : "null"; + throw new IllegalArgumentException( + String.format("Configuration step '%s' not found for connector '%s'.", configurationStepName, connectorId)); + } + + final ConfigurationStepEntity entity = new ConfigurationStepEntity(); + entity.setConfigurationStep(foundConfigurationStep); + entity.setParentConnectorId(connectorDto != null ? connectorDto.getId() : null); + entity.setParentConnectorRevision(parentConnectorRevision); + return entity; + } + + public ConnectorPropertyAllowableValuesEntity createConnectorPropertyAllowableValuesEntity( + final String configurationStepName, final String propertyGroupName, final String propertyName, final List allowableValues) { + final ConnectorPropertyAllowableValuesEntity entity = new ConnectorPropertyAllowableValuesEntity(); + entity.setConfigurationStepName(configurationStepName); + entity.setPropertyGroupName(propertyGroupName); + entity.setPropertyName(propertyName); + entity.setAllowableValues(allowableValues); + return entity; + } + + public SecretsEntity createSecretsEntity(final List secrets) { + final SecretsEntity entity = new SecretsEntity(); + entity.setSecrets(secrets); + return entity; + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/configuration/WebApplicationConfiguration.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/configuration/WebApplicationConfiguration.java index 02e23dbe2121..7ed4627cfb78 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/configuration/WebApplicationConfiguration.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/configuration/WebApplicationConfiguration.java @@ -18,6 +18,7 @@ import org.apache.nifi.admin.service.AuditService; import org.apache.nifi.audit.NiFiAuditor; +import org.apache.nifi.authorization.AuthorizableLookup; import org.apache.nifi.authorization.Authorizer; import org.apache.nifi.authorization.StandardAuthorizableLookup; import org.apache.nifi.cluster.coordination.ClusterCoordinator; @@ -31,6 +32,7 @@ import org.apache.nifi.services.FlowService; import org.apache.nifi.util.NiFiProperties; import org.apache.nifi.web.ContentAccess; +import org.apache.nifi.web.NiFiConnectorWebContext; import org.apache.nifi.web.NiFiServiceFacade; import org.apache.nifi.web.NiFiServiceFacadeLock; import org.apache.nifi.web.NiFiWebConfigurationContext; @@ -44,9 +46,11 @@ import org.apache.nifi.web.api.metrics.jmx.JmxMetricsResultConverter; import org.apache.nifi.web.api.metrics.jmx.JmxMetricsService; import org.apache.nifi.web.api.metrics.jmx.StandardJmxMetricsService; +import org.apache.nifi.web.connector.StandardNiFiConnectorWebContext; import org.apache.nifi.web.controller.ControllerFacade; import org.apache.nifi.web.controller.ControllerSearchService; import org.apache.nifi.web.dao.AccessPolicyDAO; +import org.apache.nifi.web.dao.ConnectorDAO; import org.apache.nifi.web.dao.impl.ComponentDAO; import org.apache.nifi.web.revision.RevisionManager; import org.apache.nifi.web.search.query.RegexSearchQueryParser; @@ -104,6 +108,10 @@ public class WebApplicationConfiguration { private NiFiServiceFacade serviceFacade; + private ConnectorDAO connectorDAO; + + private AuthorizableLookup authorizableLookup; + public WebApplicationConfiguration( final Authorizer authorizer, final AccessPolicyDAO accessPolicyDao, @@ -145,6 +153,16 @@ public void setServiceFacade(final NiFiServiceFacade serviceFacade) { this.serviceFacade = serviceFacade; } + @Autowired + public void setConnectorDAO(final ConnectorDAO connectorDAO) { + this.connectorDAO = connectorDAO; + } + + @Autowired + public void setAuthorizableLookup(final AuthorizableLookup authorizableLookup) { + this.authorizableLookup = authorizableLookup; + } + @Bean public EntityFactory entityFactory() { return new EntityFactory(); @@ -168,6 +186,8 @@ public DtoFactory dtoFactory() { dtoFactory.setControllerServiceProvider(flowController.getControllerServiceProvider()); dtoFactory.setEntityFactory(entityFactory()); dtoFactory.setExtensionManager(extensionManager); + dtoFactory.setConnectorRepository(flowController.getConnectorRepository()); + dtoFactory.setRuntimeManifestService(runtimeManifestService); return dtoFactory; } @@ -254,6 +274,15 @@ public NiFiWebConfigurationContext nifiWebConfigurationContext() { return context; } + @Bean + public NiFiConnectorWebContext nifiConnectorWebContext() { + final StandardNiFiConnectorWebContext context = new StandardNiFiConnectorWebContext(); + context.setConnectorDAO(connectorDAO); + context.setAuthorizer(authorizer); + context.setAuthorizableLookup(authorizableLookup); + return context; + } + @Bean public ControllerFacade controllerFacade() { final ControllerFacade controllerFacade = new ControllerFacade(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/StandardNiFiConnectorWebContext.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/StandardNiFiConnectorWebContext.java new file mode 100644 index 000000000000..5b5018174c47 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/StandardNiFiConnectorWebContext.java @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector; + +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.web.NiFiConnectorWebContext; +import org.apache.nifi.web.connector.authorization.AuthorizingConnectorInvocationHandler; +import org.apache.nifi.web.connector.authorization.AuthorizingFlowContext; +import org.apache.nifi.web.connector.authorization.ConnectorAuthorizationContext; +import org.apache.nifi.web.dao.ConnectorDAO; + +import java.lang.reflect.Proxy; + +/** + * Implements the NiFiConnectorWebContext interface to provide + * Connector instances to connector custom UIs. + * + *

The returned Connector instance is wrapped in an authorization proxy that + * enforces permissions based on the {@link ConnectorWebMethod} annotation on + * the connector interface methods. Methods without this annotation cannot be + * invoked through the proxy.

+ * + *

The returned FlowContext instances are also wrapped in authorization wrappers + * that enforce read/write permissions on all operations.

+ */ +public class StandardNiFiConnectorWebContext implements NiFiConnectorWebContext { + + private ConnectorDAO connectorDAO; + private Authorizer authorizer; + private AuthorizableLookup authorizableLookup; + + @Override + @SuppressWarnings("unchecked") + public ConnectorWebContext getConnectorWebContext(final String connectorId) throws IllegalArgumentException { + final ConnectorNode connectorNode = connectorDAO.getConnector(connectorId); + if (connectorNode == null) { + throw new IllegalArgumentException("Unable to find connector with id: " + connectorId); + } + + final ConnectorAuthorizationContext authContext = new ConnectorAuthorizationContext(connectorId, authorizer, authorizableLookup); + + final T connector = (T) connectorNode.getConnector(); + final T authorizedConnectorProxy = createAuthorizingProxy(connector, connectorId); + + final FlowContext workingFlowContext = new AuthorizingFlowContext(connectorNode.getWorkingFlowContext(), authContext); + final FlowContext activeFlowContext = new AuthorizingFlowContext(connectorNode.getActiveFlowContext(), authContext); + + return new ConnectorWebContext<>(authorizedConnectorProxy, workingFlowContext, activeFlowContext); + } + + /** + * Creates a proxy around the given connector that enforces authorization + * based on {@link ConnectorWebMethod} annotations. + * + * @param the type of the connector + * @param connector the connector instance to wrap + * @param connectorId the ID of the connector + * @return a proxy that enforces authorization on method invocations + */ + @SuppressWarnings("unchecked") + private T createAuthorizingProxy(final T connector, final String connectorId) { + final AuthorizingConnectorInvocationHandler handler = new AuthorizingConnectorInvocationHandler<>( + connector, connectorId, authorizer, authorizableLookup); + + return (T) Proxy.newProxyInstance( + connector.getClass().getClassLoader(), + connector.getClass().getInterfaces(), + handler); + } + + public void setConnectorDAO(final ConnectorDAO connectorDAO) { + this.connectorDAO = connectorDAO; + } + + public void setAuthorizer(final Authorizer authorizer) { + this.authorizer = authorizer; + } + + public void setAuthorizableLookup(final AuthorizableLookup authorizableLookup) { + this.authorizableLookup = authorizableLookup; + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectionFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectionFacade.java new file mode 100644 index 000000000000..ea72b197f6b6 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectionFacade.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.DropFlowFileSummary; +import org.apache.nifi.components.connector.components.ConnectionFacade; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flowfile.FlowFile; + +import java.io.IOException; +import java.util.Objects; +import java.util.function.Predicate; + +/** + * A wrapper around {@link ConnectionFacade} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingConnectionFacade implements ConnectionFacade { + + private final ConnectionFacade delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingConnectionFacade(final ConnectionFacade delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public VersionedConnection getDefinition() { + authContext.authorizeRead(); + return delegate.getDefinition(); + } + + @Override + public QueueSize getQueueSize() { + authContext.authorizeRead(); + return delegate.getQueueSize(); + } + + @Override + public void purge() { + authContext.authorizeWrite(); + delegate.purge(); + } + + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + authContext.authorizeWrite(); + return delegate.dropFlowFiles(predicate); + } + + @Override + public String toString() { + return "AuthorizingConnectionFacade[delegate=" + delegate + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final AuthorizingConnectionFacade that = (AuthorizingConnectionFacade) o; + return Objects.equals(delegate, that.delegate); + } + + @Override + public int hashCode() { + return Objects.hashCode(delegate); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectorInvocationHandler.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectorInvocationHandler.java new file mode 100644 index 000000000000..130fa4324852 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectorInvocationHandler.java @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserUtils; +import org.apache.nifi.web.ConnectorWebMethod; + +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; + +/** + * An InvocationHandler that wraps a Connector instance and enforces authorization + * based on the {@link ConnectorWebMethod} annotation present on the invoked method. + * + *

Methods must be annotated with {@link ConnectorWebMethod} to be invokable through + * this handler. The annotation specifies whether READ or WRITE access is required. + * Methods without the annotation will result in an {@link IllegalStateException}.

+ * + * @param the type of the Connector being proxied + */ +public class AuthorizingConnectorInvocationHandler implements InvocationHandler { + + private final T delegate; + private final String connectorId; + private final Authorizer authorizer; + private final AuthorizableLookup authorizableLookup; + + /** + * Constructs an AuthorizingConnectorInvocationHandler. + * + * @param delegate the actual Connector instance to delegate method calls to + * @param connectorId the ID of the connector, used for authorization lookups + * @param authorizer the Authorizer to use for authorization checks + * @param authorizableLookup the lookup service to obtain the Authorizable for the connector + */ + public AuthorizingConnectorInvocationHandler(final T delegate, final String connectorId, + final Authorizer authorizer, final AuthorizableLookup authorizableLookup) { + this.delegate = delegate; + this.connectorId = connectorId; + this.authorizer = authorizer; + this.authorizableLookup = authorizableLookup; + } + + @Override + public Object invoke(final Object proxy, final Method method, final Object[] args) throws Throwable { + final ConnectorWebMethod annotation = findConnectorWebMethodAnnotation(method); + + if (annotation == null) { + throw new IllegalStateException(String.format( + "Method [%s] on connector [%s] is not annotated with @ConnectorWebMethod and cannot be invoked through the Connector Web Context", + method.getName(), connectorId)); + } + + final RequestAction requiredAction = mapAccessTypeToRequestAction(annotation.value()); + final Authorizable connector = authorizableLookup.getConnector(connectorId); + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + + connector.authorize(authorizer, requiredAction, user); + + try { + return method.invoke(delegate, args); + } catch (final InvocationTargetException e) { + throw e.getCause(); + } + } + + /** + * Maps the ConnectorWebMethod.AccessType to the corresponding RequestAction. + * + * @param accessType the access type from the annotation + * @return the corresponding RequestAction + */ + private RequestAction mapAccessTypeToRequestAction(final ConnectorWebMethod.AccessType accessType) { + return switch (accessType) { + case READ -> RequestAction.READ; + case WRITE -> RequestAction.WRITE; + }; + } + + /** + * Finds the ConnectorWebMethod annotation on the given method. This method searches + * the declaring class's interfaces to find the annotation, as the method parameter + * may be from the proxy class rather than the interface. + * + * @param method the method to search for the annotation + * @return the ConnectorWebMethod annotation, or null if not found + */ + private ConnectorWebMethod findConnectorWebMethodAnnotation(final Method method) { + final ConnectorWebMethod directAnnotation = method.getAnnotation(ConnectorWebMethod.class); + if (directAnnotation != null) { + return directAnnotation; + } + + for (final Class iface : delegate.getClass().getInterfaces()) { + try { + final Method interfaceMethod = iface.getMethod(method.getName(), method.getParameterTypes()); + final ConnectorWebMethod interfaceAnnotation = interfaceMethod.getAnnotation(ConnectorWebMethod.class); + if (interfaceAnnotation != null) { + return interfaceAnnotation; + } + } catch (final NoSuchMethodException ignored) { + // Method not found on this interface; continue searching other interfaces + continue; + } + } + + return null; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingControllerServiceFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingControllerServiceFacade.java new file mode 100644 index 000000000000..a85d924079f8 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingControllerServiceFacade.java @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ControllerServiceFacade; +import org.apache.nifi.components.connector.components.ControllerServiceLifecycle; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameterContext; + +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * A wrapper around {@link ControllerServiceFacade} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingControllerServiceFacade implements ControllerServiceFacade { + + private final ControllerServiceFacade delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingControllerServiceFacade(final ControllerServiceFacade delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public VersionedControllerService getDefinition() { + authContext.authorizeRead(); + return delegate.getDefinition(); + } + + @Override + public ControllerServiceLifecycle getLifecycle() { + authContext.authorizeRead(); + return new AuthorizingControllerServiceLifecycle(delegate.getLifecycle(), authContext); + } + + @Override + public List validate() { + authContext.authorizeRead(); + return delegate.validate(); + } + + @Override + public List validate(final Map propertyValues) { + authContext.authorizeRead(); + return delegate.validate(propertyValues); + } + + @Override + public List verify(final Map propertyValues, final Map variables) { + authContext.authorizeRead(); + return delegate.verify(propertyValues, variables); + } + + @Override + public List verify(final Map propertyValues, final VersionedParameterContext parameterContext, final Map variables) { + authContext.authorizeRead(); + return delegate.verify(propertyValues, parameterContext, variables); + } + + @Override + public List verify(final VersionedExternalFlow versionedExternalFlow, final Map variables) { + authContext.authorizeRead(); + return delegate.verify(versionedExternalFlow, variables); + } + + @Override + public Object invokeConnectorMethod(final String methodName, final Map arguments) throws InvocationFailedException { + authContext.authorizeWrite(); + return delegate.invokeConnectorMethod(methodName, arguments); + } + + @Override + public T invokeConnectorMethod(final String methodName, final Map arguments, final Class returnType) throws InvocationFailedException { + authContext.authorizeWrite(); + return delegate.invokeConnectorMethod(methodName, arguments, returnType); + } + + @Override + public String toString() { + return "AuthorizingControllerServiceFacade[delegate=" + delegate + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final AuthorizingControllerServiceFacade that = (AuthorizingControllerServiceFacade) o; + return Objects.equals(delegate, that.delegate); + } + + @Override + public int hashCode() { + return Objects.hashCode(delegate); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingControllerServiceLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingControllerServiceLifecycle.java new file mode 100644 index 000000000000..9c4332a45a4b --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingControllerServiceLifecycle.java @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.components.ControllerServiceLifecycle; +import org.apache.nifi.components.connector.components.ControllerServiceState; + +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +/** + * A wrapper around {@link ControllerServiceLifecycle} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingControllerServiceLifecycle implements ControllerServiceLifecycle { + + private final ControllerServiceLifecycle delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingControllerServiceLifecycle(final ControllerServiceLifecycle delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public ControllerServiceState getState() { + authContext.authorizeRead(); + return delegate.getState(); + } + + @Override + public CompletableFuture enable() { + authContext.authorizeWrite(); + return delegate.enable(); + } + + @Override + public CompletableFuture enable(final Map propertyValueOverrides) { + authContext.authorizeWrite(); + return delegate.enable(propertyValueOverrides); + } + + @Override + public CompletableFuture disable() { + authContext.authorizeWrite(); + return delegate.disable(); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingFlowContext.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingFlowContext.java new file mode 100644 index 000000000000..d03b5e71fb3e --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingFlowContext.java @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.FlowContextType; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.flow.Bundle; + +/** + * A wrapper around {@link FlowContext} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingFlowContext implements FlowContext { + + private final FlowContext delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingFlowContext(final FlowContext delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public ProcessGroupFacade getRootGroup() { + authContext.authorizeRead(); + return new AuthorizingProcessGroupFacade(delegate.getRootGroup(), authContext); + } + + @Override + public ParameterContextFacade getParameterContext() { + authContext.authorizeRead(); + return new AuthorizingParameterContextFacade(delegate.getParameterContext(), authContext); + } + + @Override + public ConnectorConfigurationContext getConfigurationContext() { + authContext.authorizeRead(); + return delegate.getConfigurationContext(); + } + + @Override + public FlowContextType getType() { + authContext.authorizeRead(); + return delegate.getType(); + } + + @Override + public Bundle getBundle() { + authContext.authorizeRead(); + return delegate.getBundle(); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingParameterContextFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingParameterContextFacade.java new file mode 100644 index 000000000000..25eb8fb117c1 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingParameterContextFacade.java @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.connector.components.ParameterValue; + +import java.io.IOException; +import java.io.InputStream; +import java.util.Collection; +import java.util.Objects; +import java.util.Set; + +/** + * A wrapper around {@link ParameterContextFacade} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingParameterContextFacade implements ParameterContextFacade { + + private final ParameterContextFacade delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingParameterContextFacade(final ParameterContextFacade delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public void updateParameters(final Collection parameterValues) { + authContext.authorizeWrite(); + delegate.updateParameters(parameterValues); + } + + @Override + public String getValue(final String parameterName) { + authContext.authorizeRead(); + return delegate.getValue(parameterName); + } + + @Override + public Set getDefinedParameterNames() { + authContext.authorizeRead(); + return delegate.getDefinedParameterNames(); + } + + @Override + public boolean isSensitive(final String parameterName) { + authContext.authorizeRead(); + return delegate.isSensitive(parameterName); + } + + @Override + public Asset createAsset(final InputStream inputStream) throws IOException { + authContext.authorizeWrite(); + return delegate.createAsset(inputStream); + } + + @Override + public String toString() { + return "AuthorizingParameterContextFacade[delegate=" + delegate + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final AuthorizingParameterContextFacade that = (AuthorizingParameterContextFacade) o; + return Objects.equals(delegate, that.delegate); + } + + @Override + public int hashCode() { + return Objects.hashCode(delegate); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessGroupFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessGroupFacade.java new file mode 100644 index 000000000000..f20e97a9becc --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessGroupFacade.java @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.DropFlowFileSummary; +import org.apache.nifi.components.connector.components.ComponentHierarchyScope; +import org.apache.nifi.components.connector.components.ConnectionFacade; +import org.apache.nifi.components.connector.components.ControllerServiceFacade; +import org.apache.nifi.components.connector.components.ControllerServiceReferenceScope; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.components.connector.components.ProcessGroupLifecycle; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; +import org.apache.nifi.controller.queue.QueueSize; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flowfile.FlowFile; + +import java.io.IOException; +import java.util.Objects; +import java.util.Set; +import java.util.function.Predicate; +import java.util.stream.Collectors; + +/** + * A wrapper around {@link ProcessGroupFacade} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingProcessGroupFacade implements ProcessGroupFacade { + + private final ProcessGroupFacade delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingProcessGroupFacade(final ProcessGroupFacade delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public VersionedProcessGroup getDefinition() { + authContext.authorizeRead(); + return delegate.getDefinition(); + } + + @Override + public ProcessorFacade getProcessor(final String id) { + authContext.authorizeRead(); + final ProcessorFacade processor = delegate.getProcessor(id); + return processor == null ? null : new AuthorizingProcessorFacade(processor, authContext); + } + + @Override + public Set getProcessors() { + authContext.authorizeRead(); + return delegate.getProcessors().stream() + .map(p -> new AuthorizingProcessorFacade(p, authContext)) + .collect(Collectors.toSet()); + } + + @Override + public ControllerServiceFacade getControllerService(final String id) { + authContext.authorizeRead(); + final ControllerServiceFacade service = delegate.getControllerService(id); + return service == null ? null : new AuthorizingControllerServiceFacade(service, authContext); + } + + @Override + public Set getControllerServices() { + authContext.authorizeRead(); + return delegate.getControllerServices().stream() + .map(s -> new AuthorizingControllerServiceFacade(s, authContext)) + .collect(Collectors.toSet()); + } + + @Override + public Set getControllerServices(final ControllerServiceReferenceScope referenceScope, final ComponentHierarchyScope hierarchyScope) { + authContext.authorizeRead(); + return delegate.getControllerServices(referenceScope, hierarchyScope).stream() + .map(s -> new AuthorizingControllerServiceFacade(s, authContext)) + .collect(Collectors.toSet()); + } + + @Override + public ConnectionFacade getConnection(final String id) { + authContext.authorizeRead(); + final ConnectionFacade connection = delegate.getConnection(id); + return connection == null ? null : new AuthorizingConnectionFacade(connection, authContext); + } + + @Override + public Set getConnections() { + authContext.authorizeRead(); + return delegate.getConnections().stream() + .map(c -> new AuthorizingConnectionFacade(c, authContext)) + .collect(Collectors.toSet()); + } + + @Override + public ProcessGroupFacade getProcessGroup(final String id) { + authContext.authorizeRead(); + final ProcessGroupFacade group = delegate.getProcessGroup(id); + return group == null ? null : new AuthorizingProcessGroupFacade(group, authContext); + } + + @Override + public Set getProcessGroups() { + authContext.authorizeRead(); + return delegate.getProcessGroups().stream() + .map(g -> new AuthorizingProcessGroupFacade(g, authContext)) + .collect(Collectors.toSet()); + } + + @Override + public QueueSize getQueueSize() { + authContext.authorizeRead(); + return delegate.getQueueSize(); + } + + @Override + public boolean isFlowEmpty() { + authContext.authorizeRead(); + return delegate.isFlowEmpty(); + } + + @Override + public StatelessGroupLifecycle getStatelessLifecycle() { + authContext.authorizeRead(); + return new AuthorizingStatelessGroupLifecycle(delegate.getStatelessLifecycle(), authContext); + } + + @Override + public ProcessGroupLifecycle getLifecycle() { + authContext.authorizeRead(); + return new AuthorizingProcessGroupLifecycle(delegate.getLifecycle(), authContext); + } + + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + authContext.authorizeWrite(); + return delegate.dropFlowFiles(predicate); + } + + @Override + public String toString() { + return "AuthorizingProcessGroupFacade[delegate=" + delegate + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final AuthorizingProcessGroupFacade that = (AuthorizingProcessGroupFacade) o; + return Objects.equals(delegate, that.delegate); + } + + @Override + public int hashCode() { + return Objects.hashCode(delegate); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessGroupLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessGroupLifecycle.java new file mode 100644 index 000000000000..1ac3267c0111 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessGroupLifecycle.java @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.components.ComponentHierarchyScope; +import org.apache.nifi.components.connector.components.ControllerServiceReferenceScope; +import org.apache.nifi.components.connector.components.ProcessGroupLifecycle; + +import java.util.Collection; +import java.util.concurrent.CompletableFuture; + +/** + * A wrapper around {@link ProcessGroupLifecycle} that enforces authorization before delegating + * to the underlying implementation. All lifecycle operations require WRITE authorization. + */ +public class AuthorizingProcessGroupLifecycle implements ProcessGroupLifecycle { + + private final ProcessGroupLifecycle delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingProcessGroupLifecycle(final ProcessGroupLifecycle delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public CompletableFuture enableControllerServices(final ControllerServiceReferenceScope referenceScope, final ComponentHierarchyScope hierarchyScope) { + authContext.authorizeWrite(); + return delegate.enableControllerServices(referenceScope, hierarchyScope); + } + + @Override + public CompletableFuture enableControllerServices(final Collection serviceIdentifiers) { + authContext.authorizeWrite(); + return delegate.enableControllerServices(serviceIdentifiers); + } + + @Override + public CompletableFuture disableControllerServices(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.disableControllerServices(scope); + } + + @Override + public CompletableFuture disableControllerServices(final Collection serviceIdentifiers) { + authContext.authorizeWrite(); + return delegate.disableControllerServices(serviceIdentifiers); + } + + @Override + public CompletableFuture startProcessors(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.startProcessors(scope); + } + + @Override + public CompletableFuture start(final ControllerServiceReferenceScope serviceReferenceScope) { + authContext.authorizeWrite(); + return delegate.start(serviceReferenceScope); + } + + @Override + public CompletableFuture stop() { + authContext.authorizeWrite(); + return delegate.stop(); + } + + @Override + public CompletableFuture stopProcessors(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.stopProcessors(scope); + } + + @Override + public CompletableFuture startPorts(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.startPorts(scope); + } + + @Override + public CompletableFuture stopPorts(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.stopPorts(scope); + } + + @Override + public CompletableFuture startRemoteProcessGroups(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.startRemoteProcessGroups(scope); + } + + @Override + public CompletableFuture stopRemoteProcessGroups(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.stopRemoteProcessGroups(scope); + } + + @Override + public CompletableFuture startStatelessGroups(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.startStatelessGroups(scope); + } + + @Override + public CompletableFuture stopStatelessGroups(final ComponentHierarchyScope scope) { + authContext.authorizeWrite(); + return delegate.stopStatelessGroups(scope); + } + + @Override + public int getActiveThreadCount() { + authContext.authorizeRead(); + return delegate.getActiveThreadCount(); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessorFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessorFacade.java new file mode 100644 index 000000000000..d392d2c2b9f2 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessorFacade.java @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.components.ProcessorLifecycle; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * A wrapper around {@link ProcessorFacade} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingProcessorFacade implements ProcessorFacade { + + private final ProcessorFacade delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingProcessorFacade(final ProcessorFacade delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public VersionedProcessor getDefinition() { + authContext.authorizeRead(); + return delegate.getDefinition(); + } + + @Override + public ProcessorLifecycle getLifecycle() { + authContext.authorizeRead(); + return new AuthorizingProcessorLifecycle(delegate.getLifecycle(), authContext); + } + + @Override + public List validate() { + authContext.authorizeRead(); + return delegate.validate(); + } + + @Override + public List validate(final Map propertyValues) { + authContext.authorizeRead(); + return delegate.validate(propertyValues); + } + + @Override + public List verify(final Map propertyValues, final Map attributes) { + authContext.authorizeRead(); + return delegate.verify(propertyValues, attributes); + } + + @Override + public List verify(final Map propertyValues, final VersionedParameterContext parameterContext, final Map attributes) { + authContext.authorizeRead(); + return delegate.verify(propertyValues, parameterContext, attributes); + } + + @Override + public List verify(final VersionedExternalFlow versionedExternalFlow, final Map attributes) { + authContext.authorizeRead(); + return delegate.verify(versionedExternalFlow, attributes); + } + + @Override + public Object invokeConnectorMethod(final String methodName, final Map arguments) throws InvocationFailedException { + authContext.authorizeWrite(); + return delegate.invokeConnectorMethod(methodName, arguments); + } + + @Override + public T invokeConnectorMethod(final String methodName, final Map arguments, final Class returnType) throws InvocationFailedException { + authContext.authorizeWrite(); + return delegate.invokeConnectorMethod(methodName, arguments, returnType); + } + + @Override + public String toString() { + return "AuthorizingProcessorFacade[delegate=" + delegate + "]"; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + final AuthorizingProcessorFacade that = (AuthorizingProcessorFacade) o; + return Objects.equals(delegate, that.delegate); + } + + @Override + public int hashCode() { + return Objects.hashCode(delegate); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessorLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessorLifecycle.java new file mode 100644 index 000000000000..562551def84f --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingProcessorLifecycle.java @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.components.ProcessorLifecycle; +import org.apache.nifi.components.connector.components.ProcessorState; + +import java.util.concurrent.CompletableFuture; + +/** + * A wrapper around {@link ProcessorLifecycle} that enforces authorization before delegating + * to the underlying implementation. + */ +public class AuthorizingProcessorLifecycle implements ProcessorLifecycle { + + private final ProcessorLifecycle delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingProcessorLifecycle(final ProcessorLifecycle delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public ProcessorState getState() { + authContext.authorizeRead(); + return delegate.getState(); + } + + @Override + public int getActiveThreadCount() { + authContext.authorizeRead(); + return delegate.getActiveThreadCount(); + } + + @Override + public void terminate() { + authContext.authorizeWrite(); + delegate.terminate(); + } + + @Override + public CompletableFuture stop() { + authContext.authorizeWrite(); + return delegate.stop(); + } + + @Override + public CompletableFuture start() { + authContext.authorizeWrite(); + return delegate.start(); + } + + @Override + public void disable() { + authContext.authorizeWrite(); + delegate.disable(); + } + + @Override + public void enable() { + authContext.authorizeWrite(); + delegate.enable(); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingStatelessGroupLifecycle.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingStatelessGroupLifecycle.java new file mode 100644 index 000000000000..bf822669a6cf --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/AuthorizingStatelessGroupLifecycle.java @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.components.connector.components.StatelessGroupLifecycle; + +import java.util.concurrent.CompletableFuture; + +/** + * A wrapper around {@link StatelessGroupLifecycle} that enforces authorization before delegating + * to the underlying implementation. All lifecycle operations require WRITE authorization. + */ +public class AuthorizingStatelessGroupLifecycle implements StatelessGroupLifecycle { + + private final StatelessGroupLifecycle delegate; + private final ConnectorAuthorizationContext authContext; + + public AuthorizingStatelessGroupLifecycle(final StatelessGroupLifecycle delegate, final ConnectorAuthorizationContext authContext) { + this.delegate = delegate; + this.authContext = authContext; + } + + @Override + public CompletableFuture start() { + authContext.authorizeWrite(); + return delegate.start(); + } + + @Override + public CompletableFuture stop() { + authContext.authorizeWrite(); + return delegate.stop(); + } + + @Override + public CompletableFuture terminate() { + authContext.authorizeWrite(); + return delegate.terminate(); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/ConnectorAuthorizationContext.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/ConnectorAuthorizationContext.java new file mode 100644 index 000000000000..00f993004736 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/connector/authorization/ConnectorAuthorizationContext.java @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserUtils; + +/** + * Holds the authorization context needed to authorize operations on a Connector's FlowContext + * and its associated facades. + */ +public class ConnectorAuthorizationContext { + + private final String connectorId; + private final Authorizer authorizer; + private final AuthorizableLookup authorizableLookup; + + public ConnectorAuthorizationContext(final String connectorId, final Authorizer authorizer, final AuthorizableLookup authorizableLookup) { + this.connectorId = connectorId; + this.authorizer = authorizer; + this.authorizableLookup = authorizableLookup; + } + + /** + * Authorizes the current user for read access to the connector. + */ + public void authorizeRead() { + authorize(RequestAction.READ); + } + + /** + * Authorizes the current user for write access to the connector. + */ + public void authorizeWrite() { + authorize(RequestAction.WRITE); + } + + private void authorize(final RequestAction action) { + final Authorizable connector = authorizableLookup.getConnector(connectorId); + final NiFiUser user = NiFiUserUtils.getNiFiUser(); + connector.authorize(authorizer, action, user); + } + + public String getConnectorId() { + return connectorId; + } + + public Authorizer getAuthorizer() { + return authorizer; + } + + public AuthorizableLookup getAuthorizableLookup() { + return authorizableLookup; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/controller/ControllerFacade.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/controller/ControllerFacade.java index 728eec6f7956..e957880e5cd2 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/controller/ControllerFacade.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/controller/ControllerFacade.java @@ -33,6 +33,7 @@ import org.apache.nifi.bundle.Bundle; import org.apache.nifi.bundle.BundleCoordinate; import org.apache.nifi.c2.protocol.component.api.ComponentManifest; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.FlowAnalysisRuleDefinition; import org.apache.nifi.c2.protocol.component.api.FlowRegistryClientDefinition; @@ -43,6 +44,9 @@ import org.apache.nifi.cluster.protocol.NodeIdentifier; import org.apache.nifi.components.ConfigurableComponent; import org.apache.nifi.components.RequiredPermission; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.Secret; import org.apache.nifi.components.listen.ListenComponent; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Connection; @@ -493,6 +497,18 @@ public Set getFlowFileProcessorTypes(final String bundleGroup return dtoFactory.fromDocumentedTypes(getExtensionManager().getExtensions(Processor.class), bundleGroupFilter, bundleArtifactFilter, typeFilter); } + /** + * Gets the Connector types that this controller supports. + * + * @param bundleGroupFilter if specified, must be member of bundle group + * @param bundleArtifactFilter if specified, must be member of bundle artifact + * @param typeFilter if specified, type must match + * @return types + */ + public Set getConnectorTypes(final String bundleGroupFilter, final String bundleArtifactFilter, final String typeFilter) { + return dtoFactory.fromDocumentedTypes(getExtensionManager().getExtensions(Connector.class), bundleGroupFilter, bundleArtifactFilter, typeFilter); + } + /** * Gets the FlowFileComparator types that this controller supports. * @@ -664,6 +680,29 @@ public FlowAnalysisRuleDefinition getFlowAnalysisRuleDefinition(String group, St return componentManifest.getFlowAnalysisRules().stream().filter(flowAnalysisRuleDefinition -> type.equals(flowAnalysisRuleDefinition.getType())).findFirst().orElse(null); } + public ConnectorDefinition getConnectorDefinition(final String group, final String artifact, final String version, final String type) { + final ComponentManifest componentManifest = getComponentManifest(group, artifact, version); + final List connectorDefinitions = componentManifest.getConnectors(); + if (connectorDefinitions == null) { + return null; + } + final ConnectorDefinition connectorDefinition = connectorDefinitions.stream() + .filter(definition -> group.equals(definition.getGroup()) + && artifact.equals(definition.getArtifact()) + && version.equals(definition.getVersion()) + && type.equals(definition.getType())) + .findFirst() + .orElse(null); + + if (connectorDefinition != null && connectorDefinition.getConfigurationSteps() != null) { + final Map stepDocumentation = runtimeManifestService.discoverStepDocumentation(group, artifact, version, type); + final Set documentedSteps = stepDocumentation.keySet(); + connectorDefinition.getConfigurationSteps().forEach(step -> step.setDocumented(documentedSteps.contains(step.getName()))); + } + + return connectorDefinition; + } + public String getAdditionalDetails(String group, String artifact, String version, String type) { final Map additionalDetailsMap = runtimeManifestService.discoverAdditionalDetails(group, artifact, version); final File additionalDetailsFile = additionalDetailsMap.get(type); @@ -680,6 +719,22 @@ public String getAdditionalDetails(String group, String artifact, String version } } + public String getStepDocumentation(final String group, final String artifact, final String version, final String connectorType, final String stepName) { + final Map stepDocsMap = runtimeManifestService.discoverStepDocumentation(group, artifact, version, connectorType); + final File stepDocFile = stepDocsMap.get(stepName); + + if (stepDocFile == null) { + throw new ResourceNotFoundException("Unable to find step documentation for step [%s] in connector [%s]".formatted(stepName, connectorType)); + } + + try (final Stream stepDocLines = Files.lines(stepDocFile.toPath())) { + return stepDocLines.collect(Collectors.joining("\n")); + } catch (final IOException e) { + throw new RuntimeException("Unable to load step documentation content for " + + stepDocFile.getAbsolutePath() + " due to: " + e.getMessage(), e); + } + } + /** * Gets the ParameterProvider types that this controller supports. * @@ -958,6 +1013,10 @@ public void save() throws NiFiCoreException { flowService.saveFlowChanges(TimeUnit.SECONDS, writeDelaySeconds); } + public void saveImmediate() throws IOException { + flowService.saveFlowChanges(); + } + /** * Returns the socket port that the local instance is listening on for * Site-to-Site communications @@ -1013,6 +1072,9 @@ public List getResources() { resources.add(ResourceFactory.getResourceResource()); resources.add(ResourceFactory.getSiteToSiteResource()); resources.add(ResourceFactory.getParameterContextsResource()); + resources.add(ResourceFactory.getConnectorsResource()); + resources.add(ResourceFactory.getDataResource(ResourceFactory.getConnectorsResource())); + resources.add(ResourceFactory.getProvenanceDataResource(ResourceFactory.getConnectorsResource())); // add each parameter context flowController.getFlowManager().getParameterContextManager().getParameterContexts().forEach(parameterContext -> resources.add(parameterContext.getResource())); @@ -1128,6 +1190,16 @@ public List getResources() { resources.add(ResourceFactory.getOperationResource(flowRegistryResource)); } + // add each connector + for (final ConnectorNode connector : flowController.getFlowManager().getAllConnectors()) { + final Resource connectorResource = connector.getResource(); + resources.add(connectorResource); + resources.add(ResourceFactory.getDataResource(connectorResource)); + resources.add(ResourceFactory.getProvenanceDataResource(connectorResource)); + resources.add(ResourceFactory.getPolicyResource(connectorResource)); + resources.add(ResourceFactory.getOperationResource(connectorResource)); + } + return resources; } @@ -1787,11 +1859,12 @@ private ProvenanceEventDTO createProvenanceEventDto(final ProvenanceEventRecord private void setComponentDetails(final ProvenanceEventDTO dto) { final NiFiUser user = NiFiUserUtils.getNiFiUser(); - final ProcessGroup root = getRootGroup(); final Connectable connectable = findLocalConnectable(dto.getComponentId()); if (connectable != null) { - dto.setGroupId(connectable.getProcessGroup().getIdentifier()); + final ProcessGroup connectableGroup = connectable.getProcessGroup(); + dto.setGroupId(connectableGroup.getIdentifier()); + connectableGroup.getConnectorIdentifier().ifPresent(dto::setConnectorId); // if the user is approved for this component policy, provide additional details, otherwise override/redact as necessary if (Result.Approved.equals(connectable.checkAuthorization(authorizer, RequestAction.READ, user).getResult())) { @@ -1805,9 +1878,11 @@ private void setComponentDetails(final ProvenanceEventDTO dto) { return; } - final RemoteGroupPort remoteGroupPort = root.findRemoteGroupPort(dto.getComponentId()); + final RemoteGroupPort remoteGroupPort = flowController.findRemoteGroupPortIncludingConnectorManaged(dto.getComponentId()); if (remoteGroupPort != null) { - dto.setGroupId(remoteGroupPort.getProcessGroupIdentifier()); + final ProcessGroup remotePortGroup = remoteGroupPort.getProcessGroup(); + dto.setGroupId(remotePortGroup.getIdentifier()); + remotePortGroup.getConnectorIdentifier().ifPresent(dto::setConnectorId); // if the user is approved for this component policy, provide additional details, otherwise override/redact as necessary if (Result.Approved.equals(remoteGroupPort.checkAuthorization(authorizer, RequestAction.READ, user).getResult())) { @@ -1820,9 +1895,11 @@ private void setComponentDetails(final ProvenanceEventDTO dto) { return; } - final Connection connection = root.findConnection(dto.getComponentId()); + final Connection connection = flowController.findConnectionIncludingConnectorManaged(dto.getComponentId()); if (connection != null) { - dto.setGroupId(connection.getProcessGroup().getIdentifier()); + final ProcessGroup connectionGroup = connection.getProcessGroup(); + dto.setGroupId(connectionGroup.getIdentifier()); + connectionGroup.getConnectorIdentifier().ifPresent(dto::setConnectorId); // if the user is approved for this component policy, provide additional details, otherwise override/redact as necessary if (Result.Approved.equals(connection.checkAuthorization(authorizer, RequestAction.READ, user).getResult())) { @@ -1917,6 +1994,24 @@ public Set getListenPorts(final NiFiUser user) { return listenPorts; } + /** + * Searches within a connector's encapsulated process group for the specified term. + * + * @param searchLiteral search string specified by the user + * @param connectorProcessGroup the connector's managed process group to search within + * @return result + */ + public SearchResultsDTO searchConnector(final String searchLiteral, final ProcessGroup connectorProcessGroup) { + final SearchResultsDTO results = new SearchResultsDTO(); + final SearchQuery searchQuery = searchQueryParser.parse(searchLiteral, NiFiUserUtils.getNiFiUser(), connectorProcessGroup, connectorProcessGroup); + + if (!StringUtils.isEmpty(searchQuery.getTerm())) { + controllerSearchService.search(searchQuery, results); + } + + return results; + } + public void verifyComponentTypes(VersionedProcessGroup versionedFlow) { flowController.verifyComponentTypesInSnippet(versionedFlow); } @@ -1961,6 +2056,15 @@ public VersionedReportingTaskImporter createReportingTaskImporter() { return new StandardVersionedReportingTaskImporter(flowController); } + /** + * Gets all secrets from the SecretsManager. + * + * @return list of all secrets available from all secret providers + */ + public List getAllSecrets() { + return flowController.getConnectorRepository().getSecretsManager().getAllSecrets(); + } + /* * setters */ diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectionDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectionDAO.java index 98caa34453ff..8785d670a6c8 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectionDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectionDAO.java @@ -43,6 +43,15 @@ public interface ConnectionDAO { */ Connection getConnection(String id); + /** + * Gets the specified Connection, optionally including Connector-managed ProcessGroups in the search. + * + * @param id The connection id + * @param includeConnectorManaged Whether to search Connector-managed ProcessGroups + * @return The connection + */ + Connection getConnection(String id, boolean includeConnectorManaged); + /** * Gets the specified flow file drop request. * diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectorDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectorDAO.java new file mode 100644 index 000000000000..70801f1dcdc9 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ConnectorDAO.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.dao; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorUpdateContext; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; + +import java.io.IOException; +import java.io.InputStream; +import java.util.List; +import java.util.Optional; + +public interface ConnectorDAO { + + void verifyCreate(ConnectorDTO connectorDTO); + + boolean hasConnector(String id); + + ConnectorNode getConnector(String id); + + List getConnectors(); + + ConnectorNode createConnector(String type, String id, BundleCoordinate bundleCoordinate, boolean firstTimeAdded, boolean registerLogObserver); + + void updateConnector(ConnectorDTO connectorDTO); + + void deleteConnector(String id); + + void startConnector(String id); + + void stopConnector(String id); + + void drainFlowFiles(String id); + + void cancelDrainFlowFiles(String id); + + void verifyCancelDrainFlowFile(String id); + + void verifyPurgeFlowFiles(String id); + + void purgeFlowFiles(String id, String requestor); + + void updateConnectorConfigurationStep(String id, String configurationStepName, ConfigurationStepConfigurationDTO configurationStepConfiguration); + + void applyConnectorUpdate(String id, ConnectorUpdateContext updateContext); + + void discardWorkingConfiguration(String id); + + void verifyCanVerifyConfigurationStep(String id, String configurationStepName); + + List verifyConfigurationStep(String id, String configurationStepName, ConfigurationStepConfigurationDTO configurationStepConfiguration); + + List fetchAllowableValues(String id, String stepName, String propertyName, String filter); + + void verifyCreateAsset(String id); + + Asset createAsset(String id, String assetId, String assetName, InputStream content) throws IOException; + + List getAssets(String id); + + Optional getAsset(String assetId); + +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ProcessGroupDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ProcessGroupDAO.java index df546810a6c9..1dd36a46b88b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ProcessGroupDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/ProcessGroupDAO.java @@ -58,6 +58,15 @@ public interface ProcessGroupDAO { */ ProcessGroup getProcessGroup(String groupId); + /** + * Gets the specified process group, optionally including Connector-managed ProcessGroups in the search. + * + * @param groupId The process group id + * @param includeConnectorManaged Whether to search Connector-managed ProcessGroups + * @return The process group + */ + ProcessGroup getProcessGroup(String groupId, boolean includeConnectorManaged); + /** * Gets all of the process groups. * diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/ComponentDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/ComponentDAO.java index 078644a24a2a..26f0de302520 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/ComponentDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/ComponentDAO.java @@ -63,14 +63,35 @@ protected boolean isAnyNotNull(T... objects) { * @param groupId id * @return group */ - protected ProcessGroup locateProcessGroup(FlowController flowController, String groupId) { - ProcessGroup group = flowController.getFlowManager().getGroup(groupId); + protected ProcessGroup locateProcessGroup(final FlowController flowController, final String groupId) { + return locateProcessGroup(flowController, groupId, false); + } - if (group == null) { - throw new ResourceNotFoundException(String.format("Unable to locate group with id '%s'.", groupId)); + /** + * Locates the specified ProcessGroup, optionally including Connector-managed ProcessGroups. + * + * @param flowController controller + * @param groupId id + * @param includeConnectorManaged whether to include Connector-managed ProcessGroups in the search + * @return group + */ + protected ProcessGroup locateProcessGroup(final FlowController flowController, final String groupId, final boolean includeConnectorManaged) { + // First, try to find the group in the main flow hierarchy (non-Connector groups) + ProcessGroup group = flowController.getFlowManager().getGroup(groupId, null); + + if (group != null) { + return group; + } + + // Optionally search Connector-managed ProcessGroups + if (includeConnectorManaged) { + group = flowController.getFlowManager().getGroup(groupId); + if (group != null) { + return group; + } } - return group; + throw new ResourceNotFoundException(String.format("Unable to locate group with id '%s'.", groupId)); } protected void verifyCreate(final ExtensionManager extensionManager, final String type, final BundleDTO bundle) { diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectionDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectionDAO.java index 696d8c11cf8e..47aa7a14dc8b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectionDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectionDAO.java @@ -23,6 +23,8 @@ import org.apache.nifi.authorization.resource.DataAuthorizable; import org.apache.nifi.authorization.user.NiFiUser; import org.apache.nifi.authorization.user.NiFiUserUtils; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.FrameworkFlowContext; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.ConnectableType; import org.apache.nifi.connectable.Connection; @@ -76,14 +78,33 @@ public class StandardConnectionDAO extends ComponentDAO implements ConnectionDAO private Authorizer authorizer; private Connection locateConnection(final String connectionId) { + return locateConnection(connectionId, false); + } + + private Connection locateConnection(final String connectionId, final boolean includeConnectorManaged) { + // First, search the main flow hierarchy final ProcessGroup rootGroup = flowController.getFlowManager().getRootGroup(); - final Connection connection = rootGroup.findConnection(connectionId); + Connection connection = rootGroup.findConnection(connectionId); - if (connection == null) { - throw new ResourceNotFoundException(String.format("Unable to find connection with id '%s'.", connectionId)); - } else { + if (connection != null) { return connection; } + + // Optionally search Connector-managed ProcessGroups + if (includeConnectorManaged) { + for (final ConnectorNode connector : flowController.getConnectorRepository().getConnectors()) { + final FrameworkFlowContext flowContext = connector.getActiveFlowContext(); + if (flowContext != null) { + final ProcessGroup managedGroup = flowContext.getManagedProcessGroup(); + connection = managedGroup.findConnection(connectionId); + if (connection != null) { + return connection; + } + } + } + } + + throw new ResourceNotFoundException(String.format("Unable to find connection with id '%s'.", connectionId)); } @Override @@ -97,6 +118,11 @@ public Connection getConnection(final String id) { return locateConnection(id); } + @Override + public Connection getConnection(final String id, final boolean includeConnectorManaged) { + return locateConnection(id, includeConnectorManaged); + } + @Override public Set getConnections(final String groupId) { final ProcessGroup group = locateProcessGroup(flowController, groupId); @@ -105,7 +131,7 @@ public Set getConnections(final String groupId) { @Override public DropFlowFileStatus getFlowFileDropRequest(String connectionId, String dropRequestId) { - final Connection connection = locateConnection(connectionId); + final Connection connection = locateConnection(connectionId, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final DropFlowFileStatus dropRequest = queue.getDropFlowFileStatus(dropRequestId); @@ -118,7 +144,7 @@ public DropFlowFileStatus getFlowFileDropRequest(String connectionId, String dro @Override public ListFlowFileStatus getFlowFileListingRequest(String connectionId, String listingRequestId) { - final Connection connection = locateConnection(connectionId); + final Connection connection = locateConnection(connectionId, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final ListFlowFileStatus listRequest = queue.getListFlowFileStatus(listingRequestId); @@ -132,7 +158,7 @@ public ListFlowFileStatus getFlowFileListingRequest(String connectionId, String @Override public FlowFileRecord getFlowFile(String id, String flowFileUuid) { try { - final Connection connection = locateConnection(id); + final Connection connection = locateConnection(id, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final FlowFileRecord flowFile = queue.getFlowFile(flowFileUuid); @@ -342,7 +368,7 @@ public Connection createConnection(final String groupId, final ConnectionDTO con @Override public DropFlowFileStatus createFlowFileDropRequest(String id, String dropRequestId) { - final Connection connection = locateConnection(id); + final Connection connection = locateConnection(id, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final NiFiUser user = NiFiUserUtils.getNiFiUser(); @@ -355,7 +381,7 @@ public DropFlowFileStatus createFlowFileDropRequest(String id, String dropReques @Override public ListFlowFileStatus createFlowFileListingRequest(String id, String listingRequestId) { - final Connection connection = locateConnection(id); + final Connection connection = locateConnection(id, true); final FlowFileQueue queue = connection.getFlowFileQueue(); // ensure we can list @@ -450,7 +476,7 @@ private void verifyList(final FlowFileQueue queue) { @Override public void verifyList(String id) { - final Connection connection = locateConnection(id); + final Connection connection = locateConnection(id, true); final FlowFileQueue queue = connection.getFlowFileQueue(); verifyList(queue); } @@ -623,7 +649,7 @@ public void deleteConnection(final String id) { @Override public DropFlowFileStatus deleteFlowFileDropRequest(String connectionId, String dropRequestId) { - final Connection connection = locateConnection(connectionId); + final Connection connection = locateConnection(connectionId, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final DropFlowFileStatus dropFlowFileStatus = queue.cancelDropFlowFileRequest(dropRequestId); @@ -636,7 +662,7 @@ public DropFlowFileStatus deleteFlowFileDropRequest(String connectionId, String @Override public ListFlowFileStatus deleteFlowFileListingRequest(String connectionId, String listingRequestId) { - final Connection connection = locateConnection(connectionId); + final Connection connection = locateConnection(connectionId, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final ListFlowFileStatus listFlowFileStatus = queue.cancelListFlowFileRequest(listingRequestId); @@ -652,7 +678,7 @@ public DownloadableContent getContent(String id, String flowFileUuid, String req try { final NiFiUser user = NiFiUserUtils.getNiFiUser(); - final Connection connection = locateConnection(id); + final Connection connection = locateConnection(id, true); final FlowFileQueue queue = connection.getFlowFileQueue(); final FlowFileRecord flowFile = queue.getFlowFile(flowFileUuid); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectorDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectorDAO.java new file mode 100644 index 000000000000..697e8c5de6c7 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectorDAO.java @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.dao.impl; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.connector.AssetReference; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorUpdateContext; +import org.apache.nifi.components.connector.ConnectorValueReference; +import org.apache.nifi.components.connector.ConnectorValueType; +import org.apache.nifi.components.connector.SecretReference; +import org.apache.nifi.components.connector.StepConfiguration; +import org.apache.nifi.components.connector.StringLiteralValue; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.web.NiFiCoreException; +import org.apache.nifi.web.ResourceNotFoundException; +import org.apache.nifi.web.api.dto.AssetReferenceDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.dao.ConnectorDAO; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Repository; + +import java.io.IOException; +import java.io.InputStream; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ExecutionException; +import java.util.stream.Collectors; + +@Repository +public class StandardConnectorDAO implements ConnectorDAO { + + private static final Logger logger = LoggerFactory.getLogger(StandardConnectorDAO.class); + + private FlowController flowController; + + @Autowired + public void setFlowController(final FlowController flowController) { + this.flowController = flowController; + } + + private FlowManager getFlowManager() { + return flowController.getFlowManager(); + } + + private ConnectorRepository getConnectorRepository() { + return flowController.getConnectorRepository(); + } + + @Override + public void verifyCreate(final ConnectorDTO connectorDTO) { + final String id = connectorDTO.getId(); + if (id != null) { + getConnectorRepository().verifyCreate(id); + } + } + + @Override + public boolean hasConnector(final String id) { + return getConnectorRepository().getConnector(id) != null; + } + + @Override + public ConnectorNode getConnector(final String id) { + final ConnectorNode connector = getConnectorRepository().getConnector(id); + if (connector == null) { + throw new ResourceNotFoundException("Could not find Connector with ID " + id); + } + return connector; + } + + @Override + public List getConnectors() { + return getConnectorRepository().getConnectors(); + } + + @Override + public ConnectorNode createConnector(final String type, final String id, final BundleCoordinate bundleCoordinate, final boolean firstTimeAdded, final boolean registerLogObserver) { + final FlowManager flowManager = getFlowManager(); + final ConnectorNode connector = flowManager.createConnector(type, id, bundleCoordinate, firstTimeAdded, registerLogObserver); + return connector; + } + + @Override + public void updateConnector(final ConnectorDTO connectorDTO) { + final ConnectorNode connector = getConnector(connectorDTO.getId()); + if (connectorDTO.getName() != null) { + getConnectorRepository().updateConnector(connector, connectorDTO.getName()); + } + } + + @Override + public void deleteConnector(final String id) { + getConnectorRepository().deleteAssets(id); + getConnectorRepository().removeConnector(id); + } + + @Override + public void startConnector(final String id) { + final ConnectorNode connector = getConnector(id); + getConnectorRepository().startConnector(connector); + } + + @Override + public void stopConnector(final String id) { + final ConnectorNode connector = getConnector(id); + getConnectorRepository().stopConnector(connector); + } + + @Override + public void drainFlowFiles(final String id) { + final ConnectorNode connector = getConnector(id); + connector.drainFlowFiles(); + } + + @Override + public void cancelDrainFlowFiles(final String id) { + final ConnectorNode connector = getConnector(id); + connector.cancelDrainFlowFiles(); + } + + @Override + public void verifyCancelDrainFlowFile(final String id) { + final ConnectorNode connector = getConnector(id); + connector.verifyCancelDrainFlowFiles(); + } + + @Override + public void verifyPurgeFlowFiles(final String id) { + final ConnectorNode connector = getConnector(id); + connector.verifyCanPurgeFlowFiles(); + } + + @Override + public void purgeFlowFiles(final String id, final String requestor) { + final ConnectorNode connector = getConnector(id); + try { + connector.purgeFlowFiles(requestor).get(); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Thread was interrupted while purging FlowFiles for Connector " + id, e); + } catch (final ExecutionException e) { + throw new IllegalStateException("Failed to purge FlowFiles for Connector " + id, e.getCause()); + } + } + + @Override + public void updateConnectorConfigurationStep(final String id, final String configurationStepName, final ConfigurationStepConfigurationDTO configurationStepDto) { + final ConnectorNode connector = getConnector(id); + + // Convert DTO to domain object - flatten all property groups into a single StepConfiguration + final StepConfiguration stepConfiguration = convertToStepConfiguration(configurationStepDto); + + // Update the connector configuration through the repository + try { + getConnectorRepository().configureConnector(connector, configurationStepName, stepConfiguration); + } catch (final Exception e) { + throw new IllegalStateException("Failed to update connector configuration: " + e, e); + } + } + + private StepConfiguration convertToStepConfiguration(final ConfigurationStepConfigurationDTO dto) { + final Map propertyValues = new HashMap<>(); + if (dto.getPropertyGroupConfigurations() != null) { + for (final PropertyGroupConfigurationDTO groupDto : dto.getPropertyGroupConfigurations()) { + if (groupDto.getPropertyValues() != null) { + for (final Map.Entry entry : groupDto.getPropertyValues().entrySet()) { + propertyValues.put(entry.getKey(), convertToConnectorValueReference(entry.getValue())); + } + } + } + } + return new StepConfiguration(propertyValues); + } + + private ConnectorValueReference convertToConnectorValueReference(final ConnectorValueReferenceDTO dto) { + if (dto == null) { + return null; + } + final ConnectorValueType valueType = dto.getValueType() != null ? ConnectorValueType.valueOf(dto.getValueType()) : ConnectorValueType.STRING_LITERAL; + return switch (valueType) { + case STRING_LITERAL -> new StringLiteralValue(dto.getValue()); + case ASSET_REFERENCE -> new AssetReference(convertToAssetIdentifiers(dto.getAssetReferences())); + case SECRET_REFERENCE -> new SecretReference(dto.getSecretProviderId(), dto.getSecretProviderName(), dto.getSecretName(), dto.getFullyQualifiedSecretName()); + }; + } + + private Set convertToAssetIdentifiers(final List assetReferenceDTOs) { + if (assetReferenceDTOs == null || assetReferenceDTOs.isEmpty()) { + return Collections.emptySet(); + } + return assetReferenceDTOs.stream().map(AssetReferenceDTO::getId).collect(Collectors.toSet()); + } + + @Override + public void applyConnectorUpdate(final String id, final ConnectorUpdateContext updateContext) { + final ConnectorNode connector = getConnector(id); + try { + getConnectorRepository().applyUpdate(connector, updateContext); + } catch (final Exception e) { + throw new NiFiCoreException("Failed to apply connector update: " + e, e); + } + } + + @Override + public void discardWorkingConfiguration(final String id) { + final ConnectorNode connector = getConnector(id); + getConnectorRepository().discardWorkingConfiguration(connector); + } + + @Override + public void verifyCanVerifyConfigurationStep(final String id, final String configurationStepName) { + // Verify that the connector exists + getConnector(id); + } + + @Override + public List verifyConfigurationStep(final String id, final String configurationStepName, final ConfigurationStepConfigurationDTO configurationStepDto) { + final ConnectorNode connector = getConnector(id); + getConnectorRepository().syncAssetsFromProvider(connector); + final StepConfiguration stepConfiguration = convertToStepConfiguration(configurationStepDto); + return connector.verifyConfigurationStep(configurationStepName, stepConfiguration); + } + + @Override + public List fetchAllowableValues(final String id, final String stepName, final String propertyName, final String filter) { + final ConnectorNode connector = getConnector(id); + if (filter == null || filter.isEmpty()) { + return connector.fetchAllowableValues(stepName, propertyName); + } else { + return connector.fetchAllowableValues(stepName, propertyName, filter); + } + } + + @Override + public void verifyCreateAsset(final String id) { + getConnector(id); + } + + @Override + public Asset createAsset(final String id, final String assetId, final String assetName, final InputStream content) throws IOException { + return getConnectorRepository().storeAsset(id, assetId, assetName, content); + } + + @Override + public List getAssets(final String id) { + return getConnectorRepository().getAssets(id); + } + + @Override + public Optional getAsset(final String assetId) { + return getConnectorRepository().getAsset(assetId); + } +} + + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardControllerServiceDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardControllerServiceDAO.java index d7d4d51a66ea..6c13a29d0b3e 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardControllerServiceDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardControllerServiceDAO.java @@ -473,7 +473,7 @@ public List verifyConfiguration(final String contro final ConfigurationContext configurationContext = new StandardConfigurationContext(serviceNode, properties, serviceNode.getAnnotationData(), parameterLookup, flowController.getControllerServiceProvider(), null); - final List verificationResults = serviceNode.verifyConfiguration(configurationContext, configVerificationLog, variables, extensionManager); + final List verificationResults = serviceNode.verifyConfiguration(configurationContext, configVerificationLog, variables, extensionManager, parameterLookup); final List resultsDtos = verificationResults.stream() .map(this::createConfigVerificationResultDto) .collect(Collectors.toList()); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardParameterContextDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardParameterContextDAO.java index 840c8014795e..090368245eb4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardParameterContextDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardParameterContextDAO.java @@ -350,7 +350,7 @@ public void verifyAssets(final ParameterContextDTO parameterContextDto, final Ma if (parameter != null) { final List assets = parameter.getReferencedAssets() == null ? Collections.emptyList() : parameter.getReferencedAssets(); for (final Asset asset : assets) { - if (!asset.getParameterContextIdentifier().equals(parameterContextDto.getId())) { + if (!asset.getOwnerIdentifier().equals(parameterContextDto.getId())) { throw new IllegalArgumentException(String.format("Parameter [%s] is not allowed to reference asset [%s] which does not belong to parameter context [%s]", parameter.getDescriptor().getName(), asset.getName(), parameterContextDto.getId())); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAO.java index abfcb07b316c..b6612c3126d4 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAO.java @@ -44,7 +44,7 @@ import org.apache.nifi.registry.flow.StandardVersionControlInformation; import org.apache.nifi.registry.flow.VersionControlInformation; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.remote.RemoteGroupPort; import org.apache.nifi.web.ResourceNotFoundException; import org.apache.nifi.web.api.dto.ProcessGroupDTO; @@ -120,7 +120,7 @@ public ProcessGroup createProcessGroup(String parentGroupId, ProcessGroupDTO pro @Override public boolean hasProcessGroup(String groupId) { - return flowController.getFlowManager().getGroup(groupId) != null; + return flowController.getFlowManager().getGroup(groupId, null) != null; } @Override @@ -187,6 +187,11 @@ public ProcessGroup getProcessGroup(String groupId) { return locateProcessGroup(flowController, groupId); } + @Override + public ProcessGroup getProcessGroup(String groupId, boolean includeConnectorManaged) { + return locateProcessGroup(flowController, groupId, includeConnectorManaged); + } + @Override public Set getProcessGroups(final String parentGroupId, final ProcessGroupRecursivity processGroupRecursivity) { ProcessGroup group = locateProcessGroup(flowController, parentGroupId); @@ -411,7 +416,7 @@ public void activateControllerServices(final String groupId, final ControllerSer .map(flowManager::getControllerServiceNode) .collect(Collectors.toList()); - final ProcessGroup group = flowManager.getGroup(groupId); + final ProcessGroup group = flowManager.getGroup(groupId, null); if (group == null) { throw new IllegalArgumentException("Cannot activate Controller Services with IDs " + serviceIds + " because the associated Process Group (id=" + groupId + ") could not be found"); } @@ -524,7 +529,7 @@ public ProcessGroup updateVersionControlInformation(final VersionControlInformat final String groupId = versionControlInformation.getGroupId(); final ProcessGroup group = locateProcessGroup(flowController, groupId); - final NiFiRegistryFlowMapper mapper = new NiFiRegistryFlowMapper(flowController.getExtensionManager()); + final VersionedComponentFlowMapper mapper = new VersionedComponentFlowMapper(flowController.getExtensionManager()); final InstantiatedVersionedProcessGroup flowSnapshot = mapper.mapProcessGroup(group, flowController.getControllerServiceProvider(), flowController.getFlowManager(), false); updateVersionControlInformation(group, flowSnapshot, versionControlInformation, versionedComponentMapping); @@ -616,7 +621,7 @@ public void verifyDeleteFlowRegistry(String registryId) { @Override public DropFlowFileStatus createDropAllFlowFilesRequest(String processGroupId, String dropRequestId) { - ProcessGroup processGroup = locateProcessGroup(flowController, processGroupId); + ProcessGroup processGroup = locateProcessGroup(flowController, processGroupId, true); final NiFiUser user = NiFiUserUtils.getNiFiUser(); if (user == null) { @@ -628,14 +633,14 @@ public DropFlowFileStatus createDropAllFlowFilesRequest(String processGroupId, S @Override public DropFlowFileStatus getDropAllFlowFilesRequest(String processGroupId, String dropRequestId) { - ProcessGroup processGroup = locateProcessGroup(flowController, processGroupId); + ProcessGroup processGroup = locateProcessGroup(flowController, processGroupId, true); return processGroup.getDropAllFlowFilesStatus(dropRequestId); } @Override public DropFlowFileStatus deleteDropAllFlowFilesRequest(String processGroupId, String dropRequestId) { - ProcessGroup processGroup = locateProcessGroup(flowController, processGroupId); + ProcessGroup processGroup = locateProcessGroup(flowController, processGroupId, true); return processGroup.cancelDropAllFlowFiles(dropRequestId); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessorDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessorDAO.java index 97856e9ad8b2..ebdccd1c15c6 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessorDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardProcessorDAO.java @@ -490,7 +490,8 @@ public List verifyProcessorConfiguration(final Stri final LogRepository logRepository = new NopLogRepository(); final ComponentLog configVerificationLog = new SimpleProcessLogger(processor, logRepository, new StandardLoggingContext(processor)); final ExtensionManager extensionManager = flowController.getExtensionManager(); - final List verificationResults = processor.verifyConfiguration(processContext, configVerificationLog, attributes, extensionManager); + final List verificationResults = processor.verifyConfiguration(processContext, configVerificationLog, attributes, extensionManager, + processor.getProcessGroup().getParameterContext()); final List resultsDtos = verificationResults.stream() .map(this::createConfigVerificationResultDto) diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/filter/ConnectorRequestContextFilter.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/filter/ConnectorRequestContextFilter.java new file mode 100644 index 000000000000..75d76d0a8c29 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/filter/ConnectorRequestContextFilter.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.filter; + +import jakarta.servlet.Filter; +import jakarta.servlet.FilterChain; +import jakarta.servlet.FilterConfig; +import jakarta.servlet.ServletException; +import jakarta.servlet.ServletRequest; +import jakarta.servlet.ServletResponse; +import jakarta.servlet.http.HttpServletRequest; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserUtils; +import org.apache.nifi.components.connector.ConnectorRequestContext; +import org.apache.nifi.components.connector.ConnectorRequestContextHolder; +import org.apache.nifi.components.connector.StandardConnectorRequestContext; + +import java.io.IOException; +import java.util.Collections; +import java.util.Enumeration; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Servlet filter that populates the {@link ConnectorRequestContextHolder} thread-local + * with the current HTTP request headers and authenticated {@link NiFiUser}. + * + *

This filter must be registered after the Spring Security filter chain so that the + * authenticated {@link NiFiUser} is available via the security context. The context is + * always cleared in the {@code finally} block to prevent thread-local memory leaks.

+ */ +public class ConnectorRequestContextFilter implements Filter { + + @Override + public void doFilter(final ServletRequest request, final ServletResponse response, final FilterChain chain) + throws IOException, ServletException { + try { + if (request instanceof HttpServletRequest httpServletRequest) { + final ConnectorRequestContext context = createContext(httpServletRequest); + ConnectorRequestContextHolder.setContext(context); + } + chain.doFilter(request, response); + } finally { + ConnectorRequestContextHolder.clearContext(); + } + } + + private ConnectorRequestContext createContext(final HttpServletRequest request) { + final NiFiUser nifiUser = NiFiUserUtils.getNiFiUser(); + final Map> headers = captureHeaders(request); + return new StandardConnectorRequestContext(nifiUser, headers); + } + + private Map> captureHeaders(final HttpServletRequest request) { + final Map> headers = new HashMap<>(); + final Enumeration headerNames = request.getHeaderNames(); + if (headerNames != null) { + while (headerNames.hasMoreElements()) { + final String name = headerNames.nextElement(); + final List values = Collections.list(request.getHeaders(name)); + headers.put(name, Collections.unmodifiableList(values)); + } + } + return headers; + } + + @Override + public void init(final FilterConfig filterConfig) { + } + + @Override + public void destroy() { + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/webapp/WEB-INF/web.xml b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/webapp/WEB-INF/web.xml index af90238f98f5..1025a301d541 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/webapp/WEB-INF/web.xml +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/webapp/WEB-INF/web.xml @@ -61,4 +61,12 @@ springSecurityFilterChain /* + + connectorRequestContextFilter + org.apache.nifi.web.filter.ConnectorRequestContextFilter + + + connectorRequestContextFilter + /* + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestLabelAuditor.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestLabelAuditor.java index af03389ec3c2..48605cd64fe9 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestLabelAuditor.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestLabelAuditor.java @@ -119,7 +119,7 @@ void setAuditor() { @Test void testCreateLabelAdvice() { final LabelDTO labelDto = getLabelDto(); - when(flowManager.getGroup(eq(GROUP_ID))).thenReturn(processGroup); + when(flowManager.getGroup(eq(GROUP_ID), eq(null))).thenReturn(processGroup); when(flowManager.createLabel(eq(LABEL_ID), eq(LABEL))).thenReturn(new StandardLabel(LABEL_ID, LABEL)); final Label label = labelDao.createLabel(GROUP_ID, labelDto); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessGroupAuditor.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessGroupAuditor.java index 0a12bae9bbbb..48c3a57b0e1f 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessGroupAuditor.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessGroupAuditor.java @@ -176,7 +176,7 @@ void testVerifyStartProcessGroupAuditing() { when(processGroup.findRemoteGroupPort(REMOTE_GROUP_INPUT_PORT)).thenReturn(remoteGroupInputPort); when(remoteGroupInputPort.getRemoteProcessGroup()).thenReturn(remoteProcessGroup); - when(flowManager.getGroup(eq(PG_1))).thenReturn(processGroup); + when(flowManager.getGroup(eq(PG_1), eq(null))).thenReturn(processGroup); when(flowManager.findConnectable(eq(PROC_1))).thenReturn(processor1); when(flowManager.findConnectable(eq(PROC_2))).thenReturn(processor2); when(flowManager.findConnectable(eq(INPUT_PORT))).thenReturn(inputPort); @@ -286,7 +286,7 @@ void testVerifyEnableProcessGroupAuditing() { when(processGroup.findInputPort(OUTPUT_PORT)).thenReturn(null); when(processGroup.findOutputPort(OUTPUT_PORT)).thenReturn(outputPort); - when(flowManager.getGroup(eq(PG_1))).thenReturn(processGroup); + when(flowManager.getGroup(eq(PG_1), eq(null))).thenReturn(processGroup); when(flowManager.findConnectable(eq(PROC_1))).thenReturn(processor1); when(flowManager.findConnectable(eq(PROC_2))).thenReturn(processor2); when(flowManager.findConnectable(eq(INPUT_PORT))).thenReturn(inputPort); @@ -354,7 +354,7 @@ void testVerifyEnableControllerServicesAuditing() { when(cs.getName()).thenReturn(CS_1); when(processGroup.findControllerService(eq(CS_1), eq(true), eq(true))).thenReturn(cs); - when(flowManager.getGroup(eq(PG_1))).thenReturn(processGroup); + when(flowManager.getGroup(eq(PG_1), eq(null))).thenReturn(processGroup); when(flowManager.getControllerServiceNode(eq(CS_1))).thenReturn(cs); when(flowController.getFlowManager()).thenReturn(flowManager); when(flowController.getControllerServiceProvider()).thenReturn(csProvider); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessorAuditor.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessorAuditor.java index 307d0daedf05..0603361eca1d 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessorAuditor.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/audit/TestProcessorAuditor.java @@ -147,7 +147,7 @@ void testCreateProcessorAdvice() { when(flowController.getControllerServiceProvider()).thenReturn(mock(ControllerServiceProvider.class)); when(flowController.getStateManagerProvider()).thenReturn(mockStateManagerProvider); - when(flowManager.getGroup(GROUP_ID)).thenReturn(processGroup); + when(flowManager.getGroup(GROUP_ID, null)).thenReturn(processGroup); when(flowManager.createProcessor(anyString(), anyString(), any())).thenReturn(mockProcessorNode); final Bundle bundle = getBundle(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/authorization/StandardAuthorizableLookupTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/authorization/StandardAuthorizableLookupTest.java index 116f55c72703..9a7bfc6cdced 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/authorization/StandardAuthorizableLookupTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/authorization/StandardAuthorizableLookupTest.java @@ -22,21 +22,28 @@ import org.apache.nifi.authorization.resource.DataTransferAuthorizable; import org.apache.nifi.authorization.resource.OperationAuthorizable; import org.apache.nifi.authorization.resource.ProvenanceDataAuthorizable; +import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.Connection; import org.apache.nifi.controller.FlowAnalysisRuleNode; import org.apache.nifi.controller.ProcessorNode; +import org.apache.nifi.groups.ProcessGroup; import org.apache.nifi.nar.ExtensionDiscoveringManager; import org.apache.nifi.nar.ExtensionManager; import org.apache.nifi.registry.flow.FlowRegistryClientNode; import org.apache.nifi.web.controller.ControllerFacade; +import org.apache.nifi.web.dao.ConnectionDAO; import org.apache.nifi.web.dao.FlowAnalysisRuleDAO; import org.apache.nifi.web.dao.FlowRegistryDAO; +import org.apache.nifi.web.dao.ProcessGroupDAO; import org.apache.nifi.web.dao.ProcessorDAO; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; public class StandardAuthorizableLookupTest { @@ -103,6 +110,108 @@ void testGetAuthorizableFromResourceFlowAnalysisRule() { assertEquals(flowAnalysisRuleNode, authorizable); } + @Test + void testGetConnectionWithoutIncludeConnectorManaged() { + final StandardAuthorizableLookup lookup = getLookup(); + final ConnectionDAO connectionDAO = mock(ConnectionDAO.class); + final Connection connection = mock(Connection.class); + final Connectable sourceConnectable = mock(Connectable.class); + + when(connectionDAO.getConnection(eq(COMPONENT_ID), eq(false))).thenReturn(connection); + when(connection.getSource()).thenReturn(sourceConnectable); + when(connection.getDestination()).thenReturn(sourceConnectable); + when(connection.getSourceAuthorizable()).thenReturn(sourceConnectable); + lookup.setConnectionDAO(connectionDAO); + + final ConnectionAuthorizable result = lookup.getConnection(COMPONENT_ID); + + assertNotNull(result); + verify(connectionDAO).getConnection(eq(COMPONENT_ID), eq(false)); + } + + @Test + void testGetConnectionWithIncludeConnectorManagedTrue() { + final StandardAuthorizableLookup lookup = getLookup(); + final ConnectionDAO connectionDAO = mock(ConnectionDAO.class); + final Connection connection = mock(Connection.class); + final Connectable sourceConnectable = mock(Connectable.class); + + when(connectionDAO.getConnection(eq(COMPONENT_ID), eq(true))).thenReturn(connection); + when(connection.getSource()).thenReturn(sourceConnectable); + when(connection.getDestination()).thenReturn(sourceConnectable); + when(connection.getSourceAuthorizable()).thenReturn(sourceConnectable); + lookup.setConnectionDAO(connectionDAO); + + final ConnectionAuthorizable result = lookup.getConnection(COMPONENT_ID, true); + + assertNotNull(result); + verify(connectionDAO).getConnection(eq(COMPONENT_ID), eq(true)); + } + + @Test + void testGetConnectionWithIncludeConnectorManagedFalse() { + final StandardAuthorizableLookup lookup = getLookup(); + final ConnectionDAO connectionDAO = mock(ConnectionDAO.class); + final Connection connection = mock(Connection.class); + final Connectable sourceConnectable = mock(Connectable.class); + + when(connectionDAO.getConnection(eq(COMPONENT_ID), eq(false))).thenReturn(connection); + when(connection.getSource()).thenReturn(sourceConnectable); + when(connection.getDestination()).thenReturn(sourceConnectable); + when(connection.getSourceAuthorizable()).thenReturn(sourceConnectable); + lookup.setConnectionDAO(connectionDAO); + + final ConnectionAuthorizable result = lookup.getConnection(COMPONENT_ID, false); + + assertNotNull(result); + verify(connectionDAO).getConnection(eq(COMPONENT_ID), eq(false)); + } + + @Test + void testGetProcessGroupWithoutIncludeConnectorManaged() { + final StandardAuthorizableLookup lookup = getLookup(); + final ProcessGroupDAO processGroupDAO = mock(ProcessGroupDAO.class); + final ProcessGroup processGroup = mock(ProcessGroup.class); + + when(processGroupDAO.getProcessGroup(eq(COMPONENT_ID), eq(false))).thenReturn(processGroup); + lookup.setProcessGroupDAO(processGroupDAO); + + final ProcessGroupAuthorizable result = lookup.getProcessGroup(COMPONENT_ID); + + assertNotNull(result); + verify(processGroupDAO).getProcessGroup(eq(COMPONENT_ID), eq(false)); + } + + @Test + void testGetProcessGroupWithIncludeConnectorManagedTrue() { + final StandardAuthorizableLookup lookup = getLookup(); + final ProcessGroupDAO processGroupDAO = mock(ProcessGroupDAO.class); + final ProcessGroup processGroup = mock(ProcessGroup.class); + + when(processGroupDAO.getProcessGroup(eq(COMPONENT_ID), eq(true))).thenReturn(processGroup); + lookup.setProcessGroupDAO(processGroupDAO); + + final ProcessGroupAuthorizable result = lookup.getProcessGroup(COMPONENT_ID, true); + + assertNotNull(result); + verify(processGroupDAO).getProcessGroup(eq(COMPONENT_ID), eq(true)); + } + + @Test + void testGetProcessGroupWithIncludeConnectorManagedFalse() { + final StandardAuthorizableLookup lookup = getLookup(); + final ProcessGroupDAO processGroupDAO = mock(ProcessGroupDAO.class); + final ProcessGroup processGroup = mock(ProcessGroup.class); + + when(processGroupDAO.getProcessGroup(eq(COMPONENT_ID), eq(false))).thenReturn(processGroup); + lookup.setProcessGroupDAO(processGroupDAO); + + final ProcessGroupAuthorizable result = lookup.getProcessGroup(COMPONENT_ID, false); + + assertNotNull(result); + verify(processGroupDAO).getProcessGroup(eq(COMPONENT_ID), eq(false)); + } + private StandardAuthorizableLookup getLookup() { final ExtensionManager extensionManager = mock(ExtensionDiscoveringManager.class); final ControllerFacade controllerFacade = mock(ControllerFacade.class); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/StandardNiFiServiceFacadeTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/StandardNiFiServiceFacadeTest.java index 5416a41a9f31..7186baf3425b 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/StandardNiFiServiceFacadeTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/StandardNiFiServiceFacadeTest.java @@ -36,6 +36,13 @@ import org.apache.nifi.authorization.user.NiFiUserDetails; import org.apache.nifi.authorization.user.StandardNiFiUser.Builder; import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.Secret; +import org.apache.nifi.components.connector.secrets.AuthorizableSecret; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.Counter; import org.apache.nifi.controller.FlowController; import org.apache.nifi.controller.ProcessorNode; @@ -63,6 +70,7 @@ import org.apache.nifi.history.History; import org.apache.nifi.history.HistoryQuery; import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.processor.Processor; import org.apache.nifi.registry.flow.FlowRegistryUtil; import org.apache.nifi.registry.flow.RegisteredFlowSnapshot; import org.apache.nifi.registry.flow.VersionControlInformation; @@ -75,7 +83,7 @@ import org.apache.nifi.registry.flow.diff.StaticDifferenceDescriptor; import org.apache.nifi.registry.flow.mapping.FlowMappingOptions; import org.apache.nifi.registry.flow.mapping.InstantiatedVersionedProcessGroup; -import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper; +import org.apache.nifi.registry.flow.mapping.VersionedComponentFlowMapper; import org.apache.nifi.reporting.Bulletin; import org.apache.nifi.reporting.BulletinFactory; import org.apache.nifi.reporting.ComponentType; @@ -85,6 +93,7 @@ import org.apache.nifi.util.NiFiProperties; import org.apache.nifi.validation.RuleViolation; import org.apache.nifi.validation.RuleViolationsManager; +import org.apache.nifi.web.api.dto.ComponentStateDTO; import org.apache.nifi.web.api.dto.CounterDTO; import org.apache.nifi.web.api.dto.CountersDTO; import org.apache.nifi.web.api.dto.CountersSnapshotDTO; @@ -94,6 +103,7 @@ import org.apache.nifi.web.api.dto.RemoteProcessGroupDTO; import org.apache.nifi.web.api.dto.action.HistoryDTO; import org.apache.nifi.web.api.dto.action.HistoryQueryDTO; +import org.apache.nifi.web.api.dto.search.SearchResultsDTO; import org.apache.nifi.web.api.dto.status.StatusHistoryDTO; import org.apache.nifi.web.api.entity.ActionEntity; import org.apache.nifi.web.api.entity.AffectedComponentEntity; @@ -102,10 +112,13 @@ import org.apache.nifi.web.api.entity.CopyRequestEntity; import org.apache.nifi.web.api.entity.CopyResponseEntity; import org.apache.nifi.web.api.entity.ProcessGroupEntity; +import org.apache.nifi.web.api.entity.SecretsEntity; import org.apache.nifi.web.api.entity.StatusHistoryEntity; import org.apache.nifi.web.api.entity.TenantEntity; import org.apache.nifi.web.api.entity.TenantsEntity; import org.apache.nifi.web.controller.ControllerFacade; +import org.apache.nifi.web.dao.ComponentStateDAO; +import org.apache.nifi.web.dao.ConnectorDAO; import org.apache.nifi.web.dao.ProcessGroupDAO; import org.apache.nifi.web.dao.RemoteProcessGroupDAO; import org.apache.nifi.web.dao.UserDAO; @@ -325,7 +338,7 @@ public void testGetComponentsAffectedByFlowUpdate_WithNewStatelessProcessGroup_R when(flowController.getExtensionManager()).thenReturn(extensionManager); final StandardNiFiServiceFacade serviceFacadeSpy = spy(serviceFacade); - final NiFiRegistryFlowMapper flowMapper = mock(NiFiRegistryFlowMapper.class); + final VersionedComponentFlowMapper flowMapper = mock(VersionedComponentFlowMapper.class); doReturn(flowMapper).when(serviceFacadeSpy).makeNiFiRegistryFlowMapper(extensionManager); final InstantiatedVersionedProcessGroup localRoot = new InstantiatedVersionedProcessGroup("local-root-instance", groupId); @@ -511,7 +524,7 @@ public void testCopyComponents() { // use spy to mock the make() method for generating a new flow mapper to make this testable final StandardNiFiServiceFacade serviceFacadeSpy = spy(serviceFacade); - final NiFiRegistryFlowMapper flowMapper = mock(NiFiRegistryFlowMapper.class); + final VersionedComponentFlowMapper flowMapper = mock(VersionedComponentFlowMapper.class); doReturn(flowMapper).when(serviceFacadeSpy).makeNiFiRegistryFlowMapper(eq(extensionManager), any(FlowMappingOptions.class)); final InstantiatedVersionedProcessGroup nonVersionedProcessGroup = mock(InstantiatedVersionedProcessGroup.class); @@ -641,7 +654,7 @@ public void testGetCurrentFlowSnapshotByGroupId() { // use spy to mock the make() method for generating a new flow mapper to make this testable final StandardNiFiServiceFacade serviceFacadeSpy = spy(serviceFacade); - final NiFiRegistryFlowMapper flowMapper = mock(NiFiRegistryFlowMapper.class); + final VersionedComponentFlowMapper flowMapper = mock(VersionedComponentFlowMapper.class); when(serviceFacadeSpy.makeNiFiRegistryFlowMapper(extensionManager)).thenReturn(flowMapper); final InstantiatedVersionedProcessGroup nonVersionedProcessGroup = mock(InstantiatedVersionedProcessGroup.class); @@ -701,7 +714,7 @@ public void testGetCurrentFlowSnapshotByGroupIdWithReferencedControllerServices( // use spy to mock the make() method for generating a new flow mapper to make this testable final StandardNiFiServiceFacade serviceFacadeSpy = spy(serviceFacade); - final NiFiRegistryFlowMapper flowMapper = mock(NiFiRegistryFlowMapper.class); + final VersionedComponentFlowMapper flowMapper = mock(VersionedComponentFlowMapper.class); when(serviceFacadeSpy.makeNiFiRegistryFlowMapper(extensionManager)).thenReturn(flowMapper); final InstantiatedVersionedProcessGroup nonVersionedProcessGroup = spy(new InstantiatedVersionedProcessGroup(UUID.randomUUID().toString(), UUID.randomUUID().toString())); @@ -1547,6 +1560,38 @@ public void testUpdateAllCounters() { verify(dtoFactory, times(1)).createCountersDto(any()); } + @Test + public void testSearchConnector() { + final String connectorId = "connector-id"; + final String searchQuery = "test-search"; + final String managedGroupId = "managed-group-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + serviceFacade.setConnectorDAO(connectorDAO); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.getIdentifier()).thenReturn(managedGroupId); + + final ControllerFacade controllerFacade = mock(ControllerFacade.class); + final SearchResultsDTO expectedResults = new SearchResultsDTO(); + when(controllerFacade.searchConnector(searchQuery, managedProcessGroup)).thenReturn(expectedResults); + serviceFacade.setControllerFacade(controllerFacade); + + final SearchResultsDTO results = serviceFacade.searchConnector(connectorId, searchQuery); + + assertNotNull(results); + verify(connectorDAO).getConnector(connectorId); + verify(connectorNode).getActiveFlowContext(); + verify(flowContext).getManagedProcessGroup(); + verify(controllerFacade).searchConnector(searchQuery, managedProcessGroup); + } + @Test public void testUpdateAllCountersWithEmptyCounters() { // Mock ControllerFacade to return empty list @@ -1664,4 +1709,321 @@ public void testClearBulletinsForComponentsWithEmptySet() { assertNotNull(result); assertEquals(0, result.getBulletinsCleared()); } + + @Test + public void testGetSecretsFiltersUnauthorizedSecrets() { + final Authentication authentication = new NiFiAuthenticationToken(new NiFiUserDetails(new Builder().identity(USER_1).build())); + SecurityContextHolder.getContext().setAuthentication(authentication); + + final ControllerFacade controllerFacade = mock(ControllerFacade.class); + serviceFacade.setControllerFacade(controllerFacade); + + final AuthorizableSecret authorizedSecret = mock(AuthorizableSecret.class); + when(authorizedSecret.getProviderName()).thenReturn("provider1"); + when(authorizedSecret.getGroupName()).thenReturn("group1"); + when(authorizedSecret.getName()).thenReturn("authorized-secret"); + when(authorizedSecret.getDescription()).thenReturn("An authorized secret"); + when(authorizedSecret.checkAuthorization(any(Authorizer.class), any(), any())).thenReturn(AuthorizationResult.approved()); + + final AuthorizableSecret unauthorizedSecret = mock(AuthorizableSecret.class); + when(unauthorizedSecret.getProviderName()).thenReturn("provider2"); + when(unauthorizedSecret.getGroupName()).thenReturn("group2"); + when(unauthorizedSecret.getName()).thenReturn("unauthorized-secret"); + when(unauthorizedSecret.getDescription()).thenReturn("An unauthorized secret"); + when(unauthorizedSecret.checkAuthorization(any(Authorizer.class), any(), any())).thenReturn(AuthorizationResult.denied()); + + when(controllerFacade.getAllSecrets()).thenReturn(List.of(authorizedSecret, unauthorizedSecret)); + + final SecretsEntity result = serviceFacade.getSecrets(); + + assertNotNull(result); + assertNotNull(result.getSecrets()); + assertEquals(1, result.getSecrets().size()); + assertEquals("authorized-secret", result.getSecrets().get(0).getName()); + } + + @Test + public void testGetSecretsWithNonAuthorizableSecrets() { + final Authentication authentication = new NiFiAuthenticationToken(new NiFiUserDetails(new Builder().identity(USER_1).build())); + SecurityContextHolder.getContext().setAuthentication(authentication); + + final ControllerFacade controllerFacade = mock(ControllerFacade.class); + serviceFacade.setControllerFacade(controllerFacade); + + final Secret nonAuthorizableSecret = mock(Secret.class); + when(nonAuthorizableSecret.getProviderName()).thenReturn("provider1"); + when(nonAuthorizableSecret.getGroupName()).thenReturn("group1"); + when(nonAuthorizableSecret.getName()).thenReturn("non-authorizable-secret"); + when(nonAuthorizableSecret.getDescription()).thenReturn("A non-authorizable secret"); + + when(controllerFacade.getAllSecrets()).thenReturn(List.of(nonAuthorizableSecret)); + + final SecretsEntity result = serviceFacade.getSecrets(); + + assertNotNull(result); + assertNotNull(result.getSecrets()); + assertEquals(1, result.getSecrets().size()); + assertEquals("non-authorizable-secret", result.getSecrets().get(0).getName()); + } + + @Test + public void testGetSecretsWithEmptyList() { + final Authentication authentication = new NiFiAuthenticationToken(new NiFiUserDetails(new Builder().identity(USER_1).build())); + SecurityContextHolder.getContext().setAuthentication(authentication); + + final ControllerFacade controllerFacade = mock(ControllerFacade.class); + serviceFacade.setControllerFacade(controllerFacade); + + when(controllerFacade.getAllSecrets()).thenReturn(List.of()); + + final SecretsEntity result = serviceFacade.getSecrets(); + + assertNotNull(result); + assertNotNull(result.getSecrets()); + assertTrue(result.getSecrets().isEmpty()); + } + + // ----------------- + // Connector State Tests + // ----------------- + + @Test + public void testGetConnectorProcessorState() { + final String connectorId = "connector-id"; + final String processorId = "processor-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + final ComponentStateDAO componentStateDAO = mock(ComponentStateDAO.class); + final DtoFactory dtoFactory = mock(DtoFactory.class); + serviceFacade.setConnectorDAO(connectorDAO); + serviceFacade.setComponentStateDAO(componentStateDAO); + serviceFacade.setDtoFactory(dtoFactory); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + final ProcessorNode processorNode = mock(ProcessorNode.class); + final Processor processor = mock(Processor.class); + final StateMap localStateMap = mock(StateMap.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findProcessor(processorId)).thenReturn(processorNode); + when(processorNode.getProcessor()).thenReturn(processor); + when(componentStateDAO.getState(processorNode, Scope.LOCAL)).thenReturn(localStateMap); + + final ComponentStateDTO expectedDto = new ComponentStateDTO(); + expectedDto.setComponentId(processorId); + when(dtoFactory.createComponentStateDTO(eq(processorId), eq(processor.getClass()), eq(localStateMap), any())).thenReturn(expectedDto); + + final ComponentStateDTO result = serviceFacade.getConnectorProcessorState(connectorId, processorId); + + assertNotNull(result); + assertEquals(processorId, result.getComponentId()); + verify(connectorDAO).getConnector(connectorId); + verify(managedProcessGroup).findProcessor(processorId); + verify(componentStateDAO).getState(processorNode, Scope.LOCAL); + } + + @Test + public void testGetConnectorProcessorStateNotFound() { + final String connectorId = "connector-id"; + final String processorId = "non-existent-processor-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + serviceFacade.setConnectorDAO(connectorDAO); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findProcessor(processorId)).thenReturn(null); + + assertThrows(ResourceNotFoundException.class, () -> serviceFacade.getConnectorProcessorState(connectorId, processorId)); + } + + @Test + public void testVerifyCanClearConnectorProcessorState() { + final String connectorId = "connector-id"; + final String processorId = "processor-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + serviceFacade.setConnectorDAO(connectorDAO); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + final ProcessorNode processorNode = mock(ProcessorNode.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findProcessor(processorId)).thenReturn(processorNode); + + // Should not throw + serviceFacade.verifyCanClearConnectorProcessorState(connectorId, processorId); + + verify(processorNode).verifyCanClearState(); + } + + @Test + public void testClearConnectorProcessorState() { + final String connectorId = "connector-id"; + final String processorId = "processor-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + final ComponentStateDAO componentStateDAO = mock(ComponentStateDAO.class); + final DtoFactory dtoFactory = mock(DtoFactory.class); + serviceFacade.setConnectorDAO(connectorDAO); + serviceFacade.setComponentStateDAO(componentStateDAO); + serviceFacade.setDtoFactory(dtoFactory); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + final ProcessorNode processorNode = mock(ProcessorNode.class); + final Processor processor = mock(Processor.class); + final StateMap localStateMap = mock(StateMap.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findProcessor(processorId)).thenReturn(processorNode); + when(processorNode.getProcessor()).thenReturn(processor); + when(componentStateDAO.getState(processorNode, Scope.LOCAL)).thenReturn(localStateMap); + + final ComponentStateDTO expectedDto = new ComponentStateDTO(); + expectedDto.setComponentId(processorId); + when(dtoFactory.createComponentStateDTO(eq(processorId), eq(processor.getClass()), eq(localStateMap), any())).thenReturn(expectedDto); + + final ComponentStateDTO result = serviceFacade.clearConnectorProcessorState(connectorId, processorId, null); + + assertNotNull(result); + assertEquals(processorId, result.getComponentId()); + verify(componentStateDAO).clearState(processorNode, null); + } + + @Test + public void testGetConnectorControllerServiceState() { + final String connectorId = "connector-id"; + final String controllerServiceId = "controller-service-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + final ComponentStateDAO componentStateDAO = mock(ComponentStateDAO.class); + final DtoFactory dtoFactory = mock(DtoFactory.class); + serviceFacade.setConnectorDAO(connectorDAO); + serviceFacade.setComponentStateDAO(componentStateDAO); + serviceFacade.setDtoFactory(dtoFactory); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + final ControllerServiceNode controllerServiceNode = mock(ControllerServiceNode.class); + final ControllerService controllerService = mock(ControllerService.class); + final StateMap localStateMap = mock(StateMap.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findControllerService(controllerServiceId, false, true)).thenReturn(controllerServiceNode); + when(controllerServiceNode.getControllerServiceImplementation()).thenReturn(controllerService); + when(componentStateDAO.getState(controllerServiceNode, Scope.LOCAL)).thenReturn(localStateMap); + + final ComponentStateDTO expectedDto = new ComponentStateDTO(); + expectedDto.setComponentId(controllerServiceId); + when(dtoFactory.createComponentStateDTO(eq(controllerServiceId), eq(controllerService.getClass()), eq(localStateMap), any())).thenReturn(expectedDto); + + final ComponentStateDTO result = serviceFacade.getConnectorControllerServiceState(connectorId, controllerServiceId); + + assertNotNull(result); + assertEquals(controllerServiceId, result.getComponentId()); + verify(connectorDAO).getConnector(connectorId); + verify(managedProcessGroup).findControllerService(controllerServiceId, false, true); + verify(componentStateDAO).getState(controllerServiceNode, Scope.LOCAL); + } + + @Test + public void testGetConnectorControllerServiceStateNotFound() { + final String connectorId = "connector-id"; + final String controllerServiceId = "non-existent-controller-service-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + serviceFacade.setConnectorDAO(connectorDAO); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findControllerService(controllerServiceId, false, true)).thenReturn(null); + + assertThrows(ResourceNotFoundException.class, () -> serviceFacade.getConnectorControllerServiceState(connectorId, controllerServiceId)); + } + + @Test + public void testVerifyCanClearConnectorControllerServiceState() { + final String connectorId = "connector-id"; + final String controllerServiceId = "controller-service-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + serviceFacade.setConnectorDAO(connectorDAO); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + final ControllerServiceNode controllerServiceNode = mock(ControllerServiceNode.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findControllerService(controllerServiceId, false, true)).thenReturn(controllerServiceNode); + + // Should not throw + serviceFacade.verifyCanClearConnectorControllerServiceState(connectorId, controllerServiceId); + + verify(controllerServiceNode).verifyCanClearState(); + } + + @Test + public void testClearConnectorControllerServiceState() { + final String connectorId = "connector-id"; + final String controllerServiceId = "controller-service-id"; + + final ConnectorDAO connectorDAO = mock(ConnectorDAO.class); + final ComponentStateDAO componentStateDAO = mock(ComponentStateDAO.class); + final DtoFactory dtoFactory = mock(DtoFactory.class); + serviceFacade.setConnectorDAO(connectorDAO); + serviceFacade.setComponentStateDAO(componentStateDAO); + serviceFacade.setDtoFactory(dtoFactory); + + final ConnectorNode connectorNode = mock(ConnectorNode.class); + final FrameworkFlowContext flowContext = mock(FrameworkFlowContext.class); + final ProcessGroup managedProcessGroup = mock(ProcessGroup.class); + final ControllerServiceNode controllerServiceNode = mock(ControllerServiceNode.class); + final ControllerService controllerService = mock(ControllerService.class); + final StateMap localStateMap = mock(StateMap.class); + + when(connectorDAO.getConnector(connectorId)).thenReturn(connectorNode); + when(connectorNode.getActiveFlowContext()).thenReturn(flowContext); + when(flowContext.getManagedProcessGroup()).thenReturn(managedProcessGroup); + when(managedProcessGroup.findControllerService(controllerServiceId, false, true)).thenReturn(controllerServiceNode); + when(controllerServiceNode.getControllerServiceImplementation()).thenReturn(controllerService); + when(componentStateDAO.getState(controllerServiceNode, Scope.LOCAL)).thenReturn(localStateMap); + + final ComponentStateDTO expectedDto = new ComponentStateDTO(); + expectedDto.setComponentId(controllerServiceId); + when(dtoFactory.createComponentStateDTO(eq(controllerServiceId), eq(controllerService.getClass()), eq(localStateMap), any())).thenReturn(expectedDto); + + final ComponentStateDTO result = serviceFacade.clearConnectorControllerServiceState(connectorId, controllerServiceId, null); + + assertNotNull(result); + assertEquals(controllerServiceId, result.getComponentId()); + verify(componentStateDAO).clearState(controllerServiceNode, null); + } } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestConnectorResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestConnectorResource.java new file mode 100644 index 000000000000..5d50471cf961 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestConnectorResource.java @@ -0,0 +1,748 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api; + +import jakarta.servlet.ServletContext; +import jakarta.servlet.http.HttpServletRequest; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.UriBuilder; +import jakarta.ws.rs.core.UriInfo; +import org.apache.nifi.authorization.AccessDeniedException; +import org.apache.nifi.authorization.AuthorizeAccess; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.web.NiFiServiceFacade; +import org.apache.nifi.web.Revision; +import org.apache.nifi.web.api.dto.AllowableValueDTO; +import org.apache.nifi.web.api.dto.ComponentStateDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.RevisionDTO; +import org.apache.nifi.web.api.dto.flow.ProcessGroupFlowDTO; +import org.apache.nifi.web.api.entity.AllowableValueEntity; +import org.apache.nifi.web.api.entity.ComponentStateEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; +import org.apache.nifi.web.api.entity.ConnectorRunStatusEntity; +import org.apache.nifi.web.api.entity.ControllerServiceEntity; +import org.apache.nifi.web.api.entity.ControllerServicesEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.SecretsEntity; +import org.apache.nifi.web.api.request.ClientIdParameter; +import org.apache.nifi.web.api.request.LongParameter; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.net.URI; +import java.util.List; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +public class TestConnectorResource { + + @InjectMocks + private ConnectorResource connectorResource; + + @Mock + private NiFiServiceFacade serviceFacade; + + @Mock + private Authorizer authorizer; + + @Mock + private HttpServletRequest httpServletRequest; + + @Mock + private ServletContext servletContext; + + @Mock + private NiFiProperties properties; + + @Mock + private UriInfo uriInfo; + + @Mock + private UriBuilder uriBuilder; + + @Mock + private FlowResource flowResource; + + @Mock + private ControllerServiceResource controllerServiceResource; + + private static final String CONNECTOR_ID = "test-connector-id"; + private static final String CONNECTOR_NAME = "Test Connector"; + private static final String CONNECTOR_TYPE = "TestConnectorType"; + private static final String CONFIGURATION_STEP_NAME = "test-step"; + private static final String PROPERTY_GROUP_NAME = "test-group"; + private static final String PROPERTY_NAME = "test-property"; + private static final String PROCESS_GROUP_ID = "test-process-group-id"; + private static final String PROCESSOR_ID = "test-processor-id"; + private static final String CONTROLLER_SERVICE_ID = "test-controller-service-id"; + + @BeforeEach + public void setUp() throws Exception { + lenient().when(httpServletRequest.getHeader(any())).thenReturn(null); + lenient().when(httpServletRequest.getServletContext()).thenReturn(servletContext); + lenient().when(httpServletRequest.getContextPath()).thenReturn("/nifi-api"); + lenient().when(httpServletRequest.getScheme()).thenReturn("http"); + lenient().when(httpServletRequest.getServerName()).thenReturn("localhost"); + lenient().when(httpServletRequest.getServerPort()).thenReturn(8080); + lenient().when(servletContext.getInitParameter(any())).thenReturn(null); + lenient().when(properties.isNode()).thenReturn(Boolean.FALSE); + + lenient().when(uriInfo.getBaseUriBuilder()).thenReturn(uriBuilder); + lenient().when(uriBuilder.segment(any(String[].class))).thenReturn(uriBuilder); + lenient().when(uriBuilder.build()).thenReturn(new URI("http://localhost:8080/nifi-api/connectors/" + CONNECTOR_ID)); + + connectorResource.setServiceFacade(serviceFacade); + connectorResource.setFlowResource(flowResource); + connectorResource.setControllerServiceResource(controllerServiceResource); + connectorResource.httpServletRequest = httpServletRequest; + connectorResource.properties = properties; + connectorResource.uriInfo = uriInfo; + } + + @Test + public void testGetConnector() { + final ConnectorEntity connectorEntity = createConnectorEntity(); + + when(serviceFacade.getConnector(CONNECTOR_ID)).thenReturn(connectorEntity); + + try (Response response = connectorResource.getConnector(CONNECTOR_ID)) { + assertEquals(200, response.getStatus()); + assertEquals(connectorEntity, response.getEntity()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnector(CONNECTOR_ID); + } + + @Test + public void testGetConnectorNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> connectorResource.getConnector(CONNECTOR_ID)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnector(anyString()); + } + + @Test + public void testUpdateConnector() { + final ConnectorEntity requestEntity = createConnectorEntity(); + final ConnectorEntity responseEntity = createConnectorEntity(); + + when(serviceFacade.updateConnector(any(Revision.class), any(ConnectorDTO.class))).thenReturn(responseEntity); + + try (Response response = connectorResource.updateConnector(CONNECTOR_ID, requestEntity)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).verifyUpdateConnector(any(ConnectorDTO.class)); + verify(serviceFacade).updateConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testUpdateConnectorNotAuthorized() { + final ConnectorEntity requestEntity = createConnectorEntity(); + + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> connectorResource.updateConnector(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).verifyUpdateConnector(any(ConnectorDTO.class)); + verify(serviceFacade, never()).updateConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testUpdateConnectorWithMismatchedId() { + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.getComponent().setId("different-id"); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.updateConnector(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).updateConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testUpdateConnectorWithNullEntity() { + assertThrows(IllegalArgumentException.class, () -> + connectorResource.updateConnector(CONNECTOR_ID, null)); + + verify(serviceFacade, never()).updateConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testUpdateConnectorWithNullComponent() { + final ConnectorEntity requestEntity = new ConnectorEntity(); + requestEntity.setComponent(null); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.updateConnector(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).updateConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testUpdateConnectorWithNullRevision() { + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.setRevision(null); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.updateConnector(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).updateConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testDeleteConnector() { + final ConnectorEntity responseEntity = createConnectorEntity(); + + when(serviceFacade.deleteConnector(any(Revision.class), eq(CONNECTOR_ID))).thenReturn(responseEntity); + + try (Response response = connectorResource.deleteConnector(new LongParameter("1"), new ClientIdParameter("client-id"), false, CONNECTOR_ID)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).verifyDeleteConnector(CONNECTOR_ID); + verify(serviceFacade).deleteConnector(any(Revision.class), eq(CONNECTOR_ID)); + } + + @Test + public void testDeleteConnectorNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.deleteConnector(new LongParameter("1"), new ClientIdParameter("client-id"), false, CONNECTOR_ID)); + + verify(serviceFacade, never()).verifyDeleteConnector(anyString()); + verify(serviceFacade, never()).deleteConnector(any(Revision.class), anyString()); + } + + @Test + public void testUpdateRunStatus() { + final ConnectorRunStatusEntity requestEntity = createConnectorRunStatusEntity(); + final ConnectorEntity responseEntity = createConnectorEntity(); + + when(serviceFacade.scheduleConnector(any(Revision.class), eq(CONNECTOR_ID), eq(ScheduledState.RUNNING))) + .thenReturn(responseEntity); + + try (Response response = connectorResource.updateRunStatus(CONNECTOR_ID, requestEntity)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).verifyUpdateConnector(any(ConnectorDTO.class)); + verify(serviceFacade).scheduleConnector(any(Revision.class), eq(CONNECTOR_ID), eq(ScheduledState.RUNNING)); + } + + @Test + public void testUpdateRunStatusNotAuthorized() { + final ConnectorRunStatusEntity requestEntity = createConnectorRunStatusEntity(); + + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.updateRunStatus(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).verifyUpdateConnector(any(ConnectorDTO.class)); + verify(serviceFacade, never()).scheduleConnector(any(Revision.class), anyString(), any(ScheduledState.class)); + } + + @Test + public void testUpdateRunStatusWithNullEntity() { + assertThrows(IllegalArgumentException.class, () -> + connectorResource.updateRunStatus(CONNECTOR_ID, null)); + + verify(serviceFacade, never()).scheduleConnector(any(Revision.class), anyString(), any(ScheduledState.class)); + } + + @Test + public void testUpdateRunStatusWithNullRevision() { + final ConnectorRunStatusEntity requestEntity = createConnectorRunStatusEntity(); + requestEntity.setRevision(null); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.updateRunStatus(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).scheduleConnector(any(Revision.class), anyString(), any(ScheduledState.class)); + } + + @Test + public void testGetConnectorPropertyAllowableValues() { + final ConnectorPropertyAllowableValuesEntity responseEntity = createConnectorPropertyAllowableValuesEntity(); + + when(serviceFacade.getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, null)) + .thenReturn(responseEntity); + + try (Response response = connectorResource.getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, null)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, null); + } + + @Test + public void testGetConnectorPropertyAllowableValuesWithFilter() { + final String filter = "test-filter"; + final ConnectorPropertyAllowableValuesEntity responseEntity = createConnectorPropertyAllowableValuesEntity(); + + when(serviceFacade.getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, filter)) + .thenReturn(responseEntity); + + try (Response response = connectorResource.getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, filter)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, filter); + } + + @Test + public void testGetConnectorPropertyAllowableValuesNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.getConnectorPropertyAllowableValues(CONNECTOR_ID, CONFIGURATION_STEP_NAME, PROPERTY_GROUP_NAME, PROPERTY_NAME, null)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnectorPropertyAllowableValues(anyString(), anyString(), anyString(), anyString(), any()); + } + + @Test + public void testGetSecrets() { + final SecretsEntity responseEntity = new SecretsEntity(); + responseEntity.setSecrets(List.of()); + + when(serviceFacade.getSecrets()).thenReturn(responseEntity); + + try (Response response = connectorResource.getSecrets(CONNECTOR_ID)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getSecrets(); + } + + @Test + public void testGetSecretsNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> connectorResource.getSecrets(CONNECTOR_ID)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getSecrets(); + } + + @Test + public void testGetFlow() { + final ProcessGroupFlowEntity responseEntity = createProcessGroupFlowEntity(); + when(serviceFacade.getConnectorFlow(CONNECTOR_ID, PROCESS_GROUP_ID, false)).thenReturn(responseEntity); + + try (Response response = connectorResource.getFlow(CONNECTOR_ID, PROCESS_GROUP_ID, false)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectorFlow(CONNECTOR_ID, PROCESS_GROUP_ID, false); + } + + @Test + public void testGetFlowNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> connectorResource.getFlow(CONNECTOR_ID, PROCESS_GROUP_ID, false)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnectorFlow(anyString(), anyString(), eq(false)); + } + + @Test + public void testGetControllerServicesFromConnectorProcessGroup() { + final Set controllerServices = Set.of(); + when(serviceFacade.getConnectorControllerServices(CONNECTOR_ID, PROCESS_GROUP_ID, true, false, true)).thenReturn(controllerServices); + + try (Response response = connectorResource.getControllerServicesFromConnectorProcessGroup(CONNECTOR_ID, PROCESS_GROUP_ID, true, false, true)) { + assertEquals(200, response.getStatus()); + final ControllerServicesEntity entity = (ControllerServicesEntity) response.getEntity(); + assertEquals(controllerServices, entity.getControllerServices()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectorControllerServices(CONNECTOR_ID, PROCESS_GROUP_ID, true, false, true); + verify(controllerServiceResource).populateRemainingControllerServiceEntitiesContent(controllerServices); + } + + @Test + public void testGetControllerServicesFromConnectorProcessGroupNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.getControllerServicesFromConnectorProcessGroup(CONNECTOR_ID, PROCESS_GROUP_ID, true, false, true)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnectorControllerServices(anyString(), anyString(), eq(true), eq(false), eq(true)); + } + + @Test + public void testInitiateDrain() { + final ConnectorEntity requestEntity = createConnectorEntity(); + final ConnectorEntity responseEntity = createConnectorEntity(); + responseEntity.getComponent().setState("DRAINING"); + + when(serviceFacade.drainConnector(any(Revision.class), eq(CONNECTOR_ID))).thenReturn(responseEntity); + + try (Response response = connectorResource.initiateDrain(CONNECTOR_ID, requestEntity)) { + assertEquals(200, response.getStatus()); + assertEquals(responseEntity, response.getEntity()); + } + + verify(serviceFacade).verifyDrainConnector(CONNECTOR_ID); + verify(serviceFacade).drainConnector(any(Revision.class), eq(CONNECTOR_ID)); + } + + @Test + public void testInitiateDrainNotAuthorized() { + final ConnectorEntity requestEntity = createConnectorEntity(); + + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.initiateDrain(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).verifyDrainConnector(anyString()); + verify(serviceFacade, never()).drainConnector(any(Revision.class), anyString()); + } + + @Test + public void testInitiateDrainWithNullEntity() { + assertThrows(IllegalArgumentException.class, () -> + connectorResource.initiateDrain(CONNECTOR_ID, null)); + + verify(serviceFacade, never()).drainConnector(any(Revision.class), anyString()); + } + + @Test + public void testInitiateDrainWithNullRevision() { + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.setRevision(null); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.initiateDrain(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).drainConnector(any(Revision.class), anyString()); + } + + @Test + public void testInitiateDrainWithMismatchedId() { + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.setId("different-id"); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.initiateDrain(CONNECTOR_ID, requestEntity)); + + verify(serviceFacade, never()).drainConnector(any(Revision.class), anyString()); + } + + private ConnectorEntity createConnectorEntity() { + final ConnectorEntity entity = new ConnectorEntity(); + + final ConnectorDTO dto = new ConnectorDTO(); + dto.setId(CONNECTOR_ID); + dto.setName(CONNECTOR_NAME); + dto.setType(CONNECTOR_TYPE); + dto.setState("STOPPED"); + entity.setComponent(dto); + + final RevisionDTO revision = new RevisionDTO(); + revision.setVersion(1L); + revision.setClientId("client-id"); + entity.setRevision(revision); + + return entity; + } + + private ConnectorRunStatusEntity createConnectorRunStatusEntity() { + final ConnectorRunStatusEntity entity = new ConnectorRunStatusEntity(); + entity.setState("RUNNING"); + + final RevisionDTO revision = new RevisionDTO(); + revision.setVersion(1L); + revision.setClientId("client-id"); + entity.setRevision(revision); + + return entity; + } + + private ConnectorPropertyAllowableValuesEntity createConnectorPropertyAllowableValuesEntity() { + final ConnectorPropertyAllowableValuesEntity entity = new ConnectorPropertyAllowableValuesEntity(); + entity.setConfigurationStepName(CONFIGURATION_STEP_NAME); + entity.setPropertyGroupName(PROPERTY_GROUP_NAME); + entity.setPropertyName(PROPERTY_NAME); + + final AllowableValueDTO allowableValueDto1 = new AllowableValueDTO(); + allowableValueDto1.setValue("value1"); + allowableValueDto1.setDisplayName("Value 1"); + allowableValueDto1.setDescription("First allowable value"); + + final AllowableValueEntity allowableValueEntity1 = new AllowableValueEntity(); + allowableValueEntity1.setAllowableValue(allowableValueDto1); + allowableValueEntity1.setCanRead(true); + + final AllowableValueDTO allowableValueDto2 = new AllowableValueDTO(); + allowableValueDto2.setValue("value2"); + allowableValueDto2.setDisplayName("Value 2"); + allowableValueDto2.setDescription("Second allowable value"); + + final AllowableValueEntity allowableValueEntity2 = new AllowableValueEntity(); + allowableValueEntity2.setAllowableValue(allowableValueDto2); + allowableValueEntity2.setCanRead(true); + + entity.setAllowableValues(List.of(allowableValueEntity1, allowableValueEntity2)); + + return entity; + } + + private ProcessGroupFlowEntity createProcessGroupFlowEntity() { + final ProcessGroupFlowEntity entity = new ProcessGroupFlowEntity(); + final ProcessGroupFlowDTO flowDTO = new ProcessGroupFlowDTO(); + flowDTO.setId("root-process-group-id"); + entity.setProcessGroupFlow(flowDTO); + return entity; + } + + @Test + public void testCreateConnectorWithValidClientSpecifiedUuid() { + final String uppercaseUuid = "A1B2C3D4-E5F6-7890-ABCD-EF1234567890"; + final String normalizedUuid = uppercaseUuid.toLowerCase(); + + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.getComponent().setId(uppercaseUuid); + requestEntity.getComponent().setType(CONNECTOR_TYPE); + requestEntity.getRevision().setVersion(0L); + + final ConnectorEntity responseEntity = createConnectorEntity(); + responseEntity.getComponent().setId(normalizedUuid); + + when(serviceFacade.createConnector(any(Revision.class), any(ConnectorDTO.class))).thenReturn(responseEntity); + + try (Response response = connectorResource.createConnector(requestEntity)) { + assertEquals(201, response.getStatus()); + final ConnectorEntity entity = (ConnectorEntity) response.getEntity(); + assertEquals(normalizedUuid, entity.getComponent().getId()); + } + + verify(serviceFacade).verifyCreateConnector(any(ConnectorDTO.class)); + verify(serviceFacade).createConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testCreateConnectorWithInvalidClientSpecifiedUuid() { + final String invalidId = "not-a-valid-uuid"; + + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.getComponent().setId(invalidId); + requestEntity.getComponent().setType(CONNECTOR_TYPE); + requestEntity.getRevision().setVersion(0L); + + final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, () -> + connectorResource.createConnector(requestEntity)); + + assertEquals("ID [" + invalidId + "] is not a valid UUID.", exception.getMessage()); + + verify(serviceFacade, never()).createConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testCreateConnectorWithNullEntity() { + assertThrows(IllegalArgumentException.class, () -> + connectorResource.createConnector(null)); + + verify(serviceFacade, never()).createConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testCreateConnectorWithNullComponent() { + final ConnectorEntity requestEntity = new ConnectorEntity(); + requestEntity.setComponent(null); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.createConnector(requestEntity)); + + verify(serviceFacade, never()).createConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testCreateConnectorWithInvalidRevision() { + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.getComponent().setType(CONNECTOR_TYPE); + requestEntity.getRevision().setVersion(1L); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.createConnector(requestEntity)); + + verify(serviceFacade, never()).createConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testCreateConnectorWithBlankType() { + final ConnectorEntity requestEntity = createConnectorEntity(); + requestEntity.getComponent().setType(""); + requestEntity.getRevision().setVersion(0L); + + assertThrows(IllegalArgumentException.class, () -> + connectorResource.createConnector(requestEntity)); + + verify(serviceFacade, never()).createConnector(any(Revision.class), any(ConnectorDTO.class)); + } + + @Test + public void testGetConnectorProcessorState() { + final ComponentStateDTO stateDTO = new ComponentStateDTO(); + stateDTO.setComponentId(PROCESSOR_ID); + when(serviceFacade.getConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID)).thenReturn(stateDTO); + + try (Response response = connectorResource.getConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID)) { + assertEquals(200, response.getStatus()); + final ComponentStateEntity entity = (ComponentStateEntity) response.getEntity(); + assertEquals(PROCESSOR_ID, entity.getComponentState().getComponentId()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID); + } + + @Test + public void testGetConnectorProcessorStateNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.getConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnectorProcessorState(anyString(), anyString()); + } + + @Test + public void testClearConnectorProcessorState() { + final ComponentStateDTO stateDTO = new ComponentStateDTO(); + stateDTO.setComponentId(PROCESSOR_ID); + when(serviceFacade.clearConnectorProcessorState(eq(CONNECTOR_ID), eq(PROCESSOR_ID), any())).thenReturn(stateDTO); + + try (Response response = connectorResource.clearConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID, null)) { + assertEquals(200, response.getStatus()); + final ComponentStateEntity entity = (ComponentStateEntity) response.getEntity(); + assertEquals(PROCESSOR_ID, entity.getComponentState().getComponentId()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).verifyCanClearConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID); + verify(serviceFacade).clearConnectorProcessorState(eq(CONNECTOR_ID), eq(PROCESSOR_ID), any()); + } + + @Test + public void testClearConnectorProcessorStateNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.clearConnectorProcessorState(CONNECTOR_ID, PROCESSOR_ID, null)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).verifyCanClearConnectorProcessorState(anyString(), anyString()); + verify(serviceFacade, never()).clearConnectorProcessorState(anyString(), anyString(), any()); + } + + @Test + public void testGetConnectorControllerServiceState() { + final ComponentStateDTO stateDTO = new ComponentStateDTO(); + stateDTO.setComponentId(CONTROLLER_SERVICE_ID); + when(serviceFacade.getConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID)).thenReturn(stateDTO); + + try (Response response = connectorResource.getConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID)) { + assertEquals(200, response.getStatus()); + final ComponentStateEntity entity = (ComponentStateEntity) response.getEntity(); + assertEquals(CONTROLLER_SERVICE_ID, entity.getComponentState().getComponentId()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID); + } + + @Test + public void testGetConnectorControllerServiceStateNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.getConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnectorControllerServiceState(anyString(), anyString()); + } + + @Test + public void testClearConnectorControllerServiceState() { + final ComponentStateDTO stateDTO = new ComponentStateDTO(); + stateDTO.setComponentId(CONTROLLER_SERVICE_ID); + when(serviceFacade.clearConnectorControllerServiceState(eq(CONNECTOR_ID), eq(CONTROLLER_SERVICE_ID), any())).thenReturn(stateDTO); + + try (Response response = connectorResource.clearConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID, null)) { + assertEquals(200, response.getStatus()); + final ComponentStateEntity entity = (ComponentStateEntity) response.getEntity(); + assertEquals(CONTROLLER_SERVICE_ID, entity.getComponentState().getComponentId()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).verifyCanClearConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID); + verify(serviceFacade).clearConnectorControllerServiceState(eq(CONNECTOR_ID), eq(CONTROLLER_SERVICE_ID), any()); + } + + @Test + public void testClearConnectorControllerServiceStateNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> + connectorResource.clearConnectorControllerServiceState(CONNECTOR_ID, CONTROLLER_SERVICE_ID, null)); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).verifyCanClearConnectorControllerServiceState(anyString(), anyString()); + verify(serviceFacade, never()).clearConnectorControllerServiceState(anyString(), anyString(), any()); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestDataTransferResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestDataTransferResource.java index b82cbd73977b..1b68a19f7982 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestDataTransferResource.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestDataTransferResource.java @@ -242,7 +242,7 @@ public void testExtendTransaction() throws Exception { final UriInfo uriInfo = mockUriInfo(locationUriStr); final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.extendPortTransactionTTL("input-ports", "port-id", transactionId, req, res, context, uriInfo, inputStream); @@ -273,7 +273,7 @@ public void testReceiveFlowFiles() throws Exception { final ServletContext context = null; final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.receiveFlowFiles("port-id", transactionId, req, context, inputStream); @@ -300,7 +300,7 @@ public void testReceiveZeroFlowFiles() throws Exception { final ServletContext context = null; final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.receiveFlowFiles("port-id", transactionId, req, context, inputStream); @@ -319,7 +319,7 @@ public void testCommitInputPortTransaction() { final ServletContext context = null; final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.commitInputPortTransaction(ResponseCode.CONFIRM_TRANSACTION.getCode(), "port-id", transactionId, req, context, inputStream); @@ -342,7 +342,7 @@ public void testTransferFlowFiles() { final HttpServletResponse res = null; final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.transferFlowFiles("port-id", transactionId, req, res, context, inputStream); @@ -364,7 +364,7 @@ public void testCommitOutputPortTransaction() { final ServletContext context = null; final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.commitOutputPortTransaction(ResponseCode.CONFIRM_TRANSACTION.getCode(), @@ -390,7 +390,7 @@ public void testCommitOutputPortTransactionBadChecksum() throws Exception { final ServletContext context = null; final InputStream inputStream = null; - final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null)); + final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties((String) null)); final String transactionId = transactionManager.createTransaction(); final Response response = resource.commitOutputPortTransaction(ResponseCode.CONFIRM_TRANSACTION.getCode(), @@ -408,7 +408,7 @@ private DataTransferResource getDataTransferResource() { final NiFiServiceFacade serviceFacade = mock(NiFiServiceFacade.class); final HttpFlowFileServerProtocol serverProtocol = mock(HttpFlowFileServerProtocol.class); - final DataTransferResource resource = new DataTransferResource(NiFiProperties.createBasicNiFiProperties(null)) { + final DataTransferResource resource = new DataTransferResource(NiFiProperties.createBasicNiFiProperties((String) null)) { @Override protected void authorizeDataTransfer(AuthorizableLookup lookup, ResourceType resourceType, String identifier) { } @@ -418,7 +418,7 @@ HttpFlowFileServerProtocol getHttpFlowFileServerProtocol(VersionNegotiator versi return serverProtocol; } }; - resource.setProperties(NiFiProperties.createBasicNiFiProperties(null)); + resource.setProperties(NiFiProperties.createBasicNiFiProperties((String) null)); resource.setServiceFacade(serviceFacade); return resource; } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestFlowResource.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestFlowResource.java index 5bb5447d8dfa..de1cf5c48527 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestFlowResource.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestFlowResource.java @@ -29,6 +29,8 @@ import jakarta.ws.rs.core.MediaType; import jakarta.ws.rs.core.Response; import jakarta.ws.rs.core.StreamingOutput; +import org.apache.nifi.authorization.AccessDeniedException; +import org.apache.nifi.authorization.AuthorizeAccess; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.validation.DisabledServiceValidationResult; import org.apache.nifi.connectable.Port; @@ -54,10 +56,12 @@ import org.apache.nifi.web.api.dto.DifferenceDTO; import org.apache.nifi.web.api.entity.ActivateControllerServicesEntity; import org.apache.nifi.web.api.entity.ClearBulletinsForGroupRequestEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; import org.apache.nifi.web.api.entity.FlowComparisonEntity; import org.apache.nifi.web.api.request.FlowMetricsProducer; import org.apache.nifi.web.api.request.FlowMetricsReportingStrategy; import org.jetbrains.annotations.NotNull; +import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.mockito.ArgumentCaptor; @@ -94,6 +98,7 @@ import static org.mockito.Mockito.any; import static org.mockito.Mockito.anyString; import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.doThrow; import static org.mockito.Mockito.lenient; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.never; @@ -138,7 +143,17 @@ public class TestFlowResource { private NiFiServiceFacade serviceFacade; @Mock - private NiFiProperties niFiProperties; + private NiFiProperties properties; + + @Mock + private ConnectorResource connectorResource; + + @BeforeEach + public void setUp() { + lenient().when(properties.isNode()).thenReturn(Boolean.FALSE); + resource.properties = properties; + resource.setConnectorResource(connectorResource); + } @Test public void testGetFlowMetricsProducerInvalid() { @@ -450,7 +465,7 @@ public void testActivateControllerServicesStateEnabled() { entity.setId(PROCESS_GROUP_ID); entity.setState("ENABLED"); - when(niFiProperties.isNode()).thenReturn(false); + when(properties.isNode()).thenReturn(false); resource.httpServletRequest = new MockHttpServletRequest(); final ArgumentCaptor>> revisionsCaptor = ArgumentCaptor.captor(); @@ -530,7 +545,7 @@ public void testClearBulletinsIncludesControllerServices() { entity.setFromTimestamp(Instant.now()); // components is null, so clearBulletins will gather all authorized components - when(niFiProperties.isNode()).thenReturn(false); + when(properties.isNode()).thenReturn(false); resource.httpServletRequest = new MockHttpServletRequest(); // Capture the function passed to filterComponents (called 3 times) @@ -601,6 +616,31 @@ public void testClearBulletinsIncludesControllerServices() { assertEquals(5, componentIds.size(), "Should have exactly 5 authorized components"); } + @Test + public void testGetConnectors() { + final ConnectorEntity connectorEntity = new ConnectorEntity(); + + when(serviceFacade.getConnectors()).thenReturn(Set.of(connectorEntity)); + + try (Response response = resource.getConnectors()) { + assertEquals(200, response.getStatus()); + assertNotNull(response.getEntity()); + } + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade).getConnectors(); + } + + @Test + public void testGetConnectorsNotAuthorized() { + doThrow(AccessDeniedException.class).when(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + + assertThrows(AccessDeniedException.class, () -> resource.getConnectors()); + + verify(serviceFacade).authorizeAccess(any(AuthorizeAccess.class)); + verify(serviceFacade, never()).getConnectors(); + } + private void setUpGetVersionDifference() { doReturn(getDifferences()).when(serviceFacade).getVersionDifference(anyString(), any(FlowVersionLocation.class), any(FlowVersionLocation.class)); } diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/StandardNiFiConnectorWebContextTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/StandardNiFiConnectorWebContextTest.java new file mode 100644 index 000000000000..a10a15a76196 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/StandardNiFiConnectorWebContextTest.java @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector; + +import org.apache.nifi.authorization.AccessDeniedException; +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.AuthorizationRequest; +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserDetails; +import org.apache.nifi.authorization.user.StandardNiFiUser; +import org.apache.nifi.components.connector.Connector; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.web.ConnectorWebMethod; +import org.apache.nifi.web.ConnectorWebMethod.AccessType; +import org.apache.nifi.web.NiFiConnectorWebContext.ConnectorWebContext; +import org.apache.nifi.web.connector.authorization.AuthorizingFlowContext; +import org.apache.nifi.web.dao.ConnectorDAO; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.security.core.Authentication; +import org.springframework.security.core.context.SecurityContext; +import org.springframework.security.core.context.SecurityContextHolder; + +import java.lang.reflect.Proxy; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNotSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; +import static org.mockito.Mockito.withSettings; + +@ExtendWith(MockitoExtension.class) +public class StandardNiFiConnectorWebContextTest { + + private static final String CONNECTOR_ID = "test-connector-id"; + private static final String USER_IDENTITY = "test-user"; + + @Mock + private ConnectorDAO connectorDAO; + + @Mock + private Authorizer authorizer; + + @Mock + private AuthorizableLookup authorizableLookup; + + @Mock + private ConnectorNode connectorNode; + + @Mock + private FrameworkFlowContext workingFlowContext; + + @Mock + private FrameworkFlowContext activeFlowContext; + + @Mock + private Authorizable connectorAuthorizable; + + @Mock + private SecurityContext securityContext; + + @Mock + private Authentication authentication; + + private StandardNiFiConnectorWebContext context; + + private TestConnector testConnectorMock; + private String lastWrittenValue; + + @BeforeEach + void setUp() { + context = new StandardNiFiConnectorWebContext(); + context.setConnectorDAO(connectorDAO); + context.setAuthorizer(authorizer); + context.setAuthorizableLookup(authorizableLookup); + + testConnectorMock = mock(TestConnector.class, withSettings().extraInterfaces(Connector.class).lenient()); + lenient().when(testConnectorMock.readData()).thenReturn("read-result"); + lenient().when(testConnectorMock.writeData(any())).thenAnswer(invocation -> { + lastWrittenValue = invocation.getArgument(0); + return null; + }); + + lenient().when(connectorDAO.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + lenient().when(connectorNode.getConnector()).thenReturn((Connector) testConnectorMock); + lenient().when(connectorNode.getWorkingFlowContext()).thenReturn(workingFlowContext); + lenient().when(connectorNode.getActiveFlowContext()).thenReturn(activeFlowContext); + lenient().when(authorizableLookup.getConnector(CONNECTOR_ID)).thenReturn(connectorAuthorizable); + lenient().when(authorizer.authorize(any(AuthorizationRequest.class))).thenReturn(AuthorizationResult.approved()); + + final NiFiUser user = new StandardNiFiUser.Builder().identity(USER_IDENTITY).build(); + final NiFiUserDetails userDetails = new NiFiUserDetails(user); + lenient().when(authentication.getPrincipal()).thenReturn(userDetails); + lenient().when(securityContext.getAuthentication()).thenReturn(authentication); + SecurityContextHolder.setContext(securityContext); + } + + @AfterEach + void tearDown() { + SecurityContextHolder.clearContext(); + } + + @Test + void testGetConnectorWebContextReturnsProxiedConnector() { + final ConnectorWebContext webContext = context.getConnectorWebContext(CONNECTOR_ID); + + assertNotNull(webContext); + assertNotNull(webContext.connector()); + assertTrue(Proxy.isProxyClass(webContext.connector().getClass())); + assertNotSame(testConnectorMock, webContext.connector()); + } + + @Test + void testGetConnectorWebContextReturnsWrappedFlowContexts() { + final ConnectorWebContext webContext = context.getConnectorWebContext(CONNECTOR_ID); + + assertNotNull(webContext.workingFlowContext()); + assertNotNull(webContext.activeFlowContext()); + assertTrue(webContext.workingFlowContext() instanceof AuthorizingFlowContext); + assertTrue(webContext.activeFlowContext() instanceof AuthorizingFlowContext); + } + + @Test + void testGetConnectorWebContextThrowsForNonExistentConnector() { + when(connectorDAO.getConnector("non-existent-id")).thenReturn(null); + + final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, + () -> context.getConnectorWebContext("non-existent-id")); + assertEquals("Unable to find connector with id: non-existent-id", exception.getMessage()); + } + + @Test + void testProxiedConnectorEnforcesReadAuthorization() { + final ConnectorWebContext webContext = context.getConnectorWebContext(CONNECTOR_ID); + final TestConnector proxy = webContext.connector(); + + final String result = proxy.readData(); + + assertEquals("read-result", result); + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testProxiedConnectorEnforcesWriteAuthorization() { + final ConnectorWebContext webContext = context.getConnectorWebContext(CONNECTOR_ID); + final TestConnector proxy = webContext.connector(); + + proxy.writeData("test-value"); + + assertEquals("test-value", lastWrittenValue); + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testProxiedConnectorBlocksUnannotatedMethods() { + final ConnectorWebContext webContext = context.getConnectorWebContext(CONNECTOR_ID); + final TestConnector proxy = webContext.connector(); + + assertThrows(IllegalStateException.class, proxy::unannotatedMethod); + } + + @Test + void testProxiedConnectorPropagatesAuthorizationFailure() { + doThrow(new AccessDeniedException("Access denied")) + .when(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + + final ConnectorWebContext webContext = context.getConnectorWebContext(CONNECTOR_ID); + final TestConnector proxy = webContext.connector(); + + assertThrows(AccessDeniedException.class, proxy::readData); + } + + /** + * Test interface representing a Connector with annotated methods. + * This interface is used with Mockito's extraInterfaces to create a mock + * that implements both this interface and Connector. + */ + public interface TestConnector { + + @ConnectorWebMethod(AccessType.READ) + String readData(); + + @ConnectorWebMethod(AccessType.WRITE) + Void writeData(String value); + + void unannotatedMethod(); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectorInvocationHandlerTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectorInvocationHandlerTest.java new file mode 100644 index 000000000000..e12b7366f9ed --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingConnectorInvocationHandlerTest.java @@ -0,0 +1,239 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.authorization.AccessDeniedException; +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.AuthorizationRequest; +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserDetails; +import org.apache.nifi.authorization.user.StandardNiFiUser; +import org.apache.nifi.web.ConnectorWebMethod; +import org.apache.nifi.web.ConnectorWebMethod.AccessType; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.security.core.Authentication; +import org.springframework.security.core.context.SecurityContext; +import org.springframework.security.core.context.SecurityContextHolder; + +import java.lang.reflect.Proxy; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.verify; + +@ExtendWith(MockitoExtension.class) +public class AuthorizingConnectorInvocationHandlerTest { + + private static final String CONNECTOR_ID = "test-connector-id"; + private static final String USER_IDENTITY = "test-user"; + + @Mock + private Authorizer authorizer; + + @Mock + private AuthorizableLookup authorizableLookup; + + @Mock + private Authorizable connectorAuthorizable; + + @Mock + private SecurityContext securityContext; + + @Mock + private Authentication authentication; + + private TestConnectorImpl connectorImpl; + + @BeforeEach + void setUp() { + connectorImpl = new TestConnectorImpl(); + lenient().when(authorizableLookup.getConnector(CONNECTOR_ID)).thenReturn(connectorAuthorizable); + + final NiFiUser user = new StandardNiFiUser.Builder().identity(USER_IDENTITY).build(); + final NiFiUserDetails userDetails = new NiFiUserDetails(user); + lenient().when(authentication.getPrincipal()).thenReturn(userDetails); + lenient().when(securityContext.getAuthentication()).thenReturn(authentication); + SecurityContextHolder.setContext(securityContext); + + lenient().when(authorizer.authorize(any(AuthorizationRequest.class))).thenReturn(AuthorizationResult.approved()); + } + + @AfterEach + void tearDown() { + SecurityContextHolder.clearContext(); + } + + @Test + void testReadMethodAuthorizesWithReadAction() { + final TestConnector proxy = createProxy(); + + final String result = proxy.readData(); + + assertEquals("read-result", result); + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.READ, actionCaptor.getValue()); + } + + @Test + void testWriteMethodAuthorizesWithWriteAction() { + final TestConnector proxy = createProxy(); + + proxy.writeData("test-value"); + + assertEquals("test-value", connectorImpl.getLastWrittenValue()); + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.WRITE, actionCaptor.getValue()); + } + + @Test + void testMethodWithoutAnnotationThrowsException() { + final TestConnector proxy = createProxy(); + + final IllegalStateException exception = assertThrows(IllegalStateException.class, proxy::unannotatedMethod); + assertEquals(String.format("Method [unannotatedMethod] on connector [%s] is not annotated with " + + "@ConnectorWebMethod and cannot be invoked through the Connector Web Context", CONNECTOR_ID), exception.getMessage()); + } + + @Test + void testAuthorizationFailurePropagatesAccessDeniedException() { + doThrow(new AccessDeniedException("Access denied for testing")) + .when(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + + final TestConnector proxy = createProxy(); + + assertThrows(AccessDeniedException.class, proxy::readData); + } + + @Test + void testMethodArgumentsPassedCorrectly() { + final TestConnector proxy = createProxy(); + + final int result = proxy.processItems(List.of("a", "b", "c"), 10); + + assertEquals(13, result); + } + + @Test + void testDelegateExceptionUnwrapped() { + final TestConnector proxy = createProxy(); + + final RuntimeException exception = assertThrows(RuntimeException.class, proxy::throwingMethod); + assertEquals("Intentional test exception", exception.getMessage()); + } + + @Test + void testReadMethodWithReturnValue() { + final TestConnector proxy = createProxy(); + + final List result = proxy.getItems(); + + assertEquals(List.of("item1", "item2", "item3"), result); + } + + private TestConnector createProxy() { + final AuthorizingConnectorInvocationHandler handler = new AuthorizingConnectorInvocationHandler<>( + connectorImpl, CONNECTOR_ID, authorizer, authorizableLookup); + + return (TestConnector) Proxy.newProxyInstance( + TestConnector.class.getClassLoader(), + new Class[]{TestConnector.class}, + handler); + } + + /** + * Test interface representing a Connector with annotated methods. + */ + public interface TestConnector { + + @ConnectorWebMethod(AccessType.READ) + String readData(); + + @ConnectorWebMethod(AccessType.WRITE) + void writeData(String value); + + @ConnectorWebMethod(AccessType.READ) + List getItems(); + + @ConnectorWebMethod(AccessType.WRITE) + int processItems(List items, int multiplier); + + @ConnectorWebMethod(AccessType.READ) + void throwingMethod(); + + void unannotatedMethod(); + } + + /** + * Test implementation of the TestConnector interface. + */ + public static class TestConnectorImpl implements TestConnector { + + private String lastWrittenValue; + + @Override + public String readData() { + return "read-result"; + } + + @Override + public void writeData(final String value) { + this.lastWrittenValue = value; + } + + @Override + public List getItems() { + return List.of("item1", "item2", "item3"); + } + + @Override + public int processItems(final List items, final int multiplier) { + return items.size() + multiplier; + } + + @Override + public void throwingMethod() { + throw new RuntimeException("Intentional test exception"); + } + + @Override + public void unannotatedMethod() { + // This method is intentionally not annotated + } + + public String getLastWrittenValue() { + return lastWrittenValue; + } + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingFlowContextTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingFlowContextTest.java new file mode 100644 index 000000000000..ddc4f277ba56 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingFlowContextTest.java @@ -0,0 +1,167 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.authorization.AccessDeniedException; +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.AuthorizationRequest; +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserDetails; +import org.apache.nifi.authorization.user.StandardNiFiUser; +import org.apache.nifi.components.connector.ConnectorConfigurationContext; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.FlowContextType; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.flow.Bundle; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.security.core.Authentication; +import org.springframework.security.core.context.SecurityContext; +import org.springframework.security.core.context.SecurityContextHolder; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +public class AuthorizingFlowContextTest { + + private static final String CONNECTOR_ID = "test-connector-id"; + private static final String USER_IDENTITY = "test-user"; + + @Mock + private FlowContext delegate; + + @Mock + private Authorizer authorizer; + + @Mock + private AuthorizableLookup authorizableLookup; + + @Mock + private Authorizable connectorAuthorizable; + + @Mock + private ProcessGroupFacade processGroupFacade; + + @Mock + private ParameterContextFacade parameterContextFacade; + + @Mock + private ConnectorConfigurationContext configurationContext; + + @Mock + private SecurityContext securityContext; + + @Mock + private Authentication authentication; + + private AuthorizingFlowContext authorizingFlowContext; + + @BeforeEach + void setUp() { + when(authorizableLookup.getConnector(CONNECTOR_ID)).thenReturn(connectorAuthorizable); + lenient().when(authorizer.authorize(any(AuthorizationRequest.class))).thenReturn(AuthorizationResult.approved()); + + final NiFiUser user = new StandardNiFiUser.Builder().identity(USER_IDENTITY).build(); + final NiFiUserDetails userDetails = new NiFiUserDetails(user); + when(authentication.getPrincipal()).thenReturn(userDetails); + when(securityContext.getAuthentication()).thenReturn(authentication); + SecurityContextHolder.setContext(securityContext); + + final ConnectorAuthorizationContext authContext = new ConnectorAuthorizationContext(CONNECTOR_ID, authorizer, authorizableLookup); + authorizingFlowContext = new AuthorizingFlowContext(delegate, authContext); + } + + @AfterEach + void tearDown() { + SecurityContextHolder.clearContext(); + } + + @Test + void testGetRootGroupAuthorizesReadAndReturnsWrappedFacade() { + when(delegate.getRootGroup()).thenReturn(processGroupFacade); + + final ProcessGroupFacade result = authorizingFlowContext.getRootGroup(); + + assertNotNull(result); + assertTrue(result instanceof AuthorizingProcessGroupFacade); + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testGetParameterContextAuthorizesReadAndReturnsWrappedFacade() { + when(delegate.getParameterContext()).thenReturn(parameterContextFacade); + + final ParameterContextFacade result = authorizingFlowContext.getParameterContext(); + + assertNotNull(result); + assertTrue(result instanceof AuthorizingParameterContextFacade); + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testGetConfigurationContextAuthorizesRead() { + when(delegate.getConfigurationContext()).thenReturn(configurationContext); + + final ConnectorConfigurationContext result = authorizingFlowContext.getConfigurationContext(); + + assertNotNull(result); + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testGetTypeAuthorizesRead() { + when(delegate.getType()).thenReturn(FlowContextType.WORKING); + + authorizingFlowContext.getType(); + + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testGetBundleAuthorizesRead() { + when(delegate.getBundle()).thenReturn(new Bundle("group", "artifact", "version")); + + authorizingFlowContext.getBundle(); + + verify(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + } + + @Test + void testAuthorizationFailurePropagates() { + doThrow(new AccessDeniedException("Access denied")) + .when(connectorAuthorizable).authorize(any(Authorizer.class), any(RequestAction.class), any(NiFiUser.class)); + + assertThrows(AccessDeniedException.class, () -> authorizingFlowContext.getRootGroup()); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingParameterContextFacadeTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingParameterContextFacadeTest.java new file mode 100644 index 000000000000..6fe3cdfeb332 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/connector/authorization/AuthorizingParameterContextFacadeTest.java @@ -0,0 +1,162 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.connector.authorization; + +import org.apache.nifi.asset.Asset; +import org.apache.nifi.authorization.AuthorizableLookup; +import org.apache.nifi.authorization.AuthorizationRequest; +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserDetails; +import org.apache.nifi.authorization.user.StandardNiFiUser; +import org.apache.nifi.components.connector.components.ParameterContextFacade; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.security.core.Authentication; +import org.springframework.security.core.context.SecurityContext; +import org.springframework.security.core.context.SecurityContextHolder; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.util.Collections; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +public class AuthorizingParameterContextFacadeTest { + + private static final String CONNECTOR_ID = "test-connector-id"; + private static final String USER_IDENTITY = "test-user"; + + @Mock + private ParameterContextFacade delegate; + + @Mock + private Authorizer authorizer; + + @Mock + private AuthorizableLookup authorizableLookup; + + @Mock + private Authorizable connectorAuthorizable; + + @Mock + private SecurityContext securityContext; + + @Mock + private Authentication authentication; + + @Mock + private Asset asset; + + private AuthorizingParameterContextFacade authorizingFacade; + + @BeforeEach + void setUp() { + when(authorizableLookup.getConnector(CONNECTOR_ID)).thenReturn(connectorAuthorizable); + lenient().when(authorizer.authorize(any(AuthorizationRequest.class))).thenReturn(AuthorizationResult.approved()); + + final NiFiUser user = new StandardNiFiUser.Builder().identity(USER_IDENTITY).build(); + final NiFiUserDetails userDetails = new NiFiUserDetails(user); + when(authentication.getPrincipal()).thenReturn(userDetails); + when(securityContext.getAuthentication()).thenReturn(authentication); + SecurityContextHolder.setContext(securityContext); + + final ConnectorAuthorizationContext authContext = new ConnectorAuthorizationContext(CONNECTOR_ID, authorizer, authorizableLookup); + authorizingFacade = new AuthorizingParameterContextFacade(delegate, authContext); + } + + @AfterEach + void tearDown() { + SecurityContextHolder.clearContext(); + } + + @Test + void testGetValueAuthorizesWithReadAction() { + when(delegate.getValue("param1")).thenReturn("value1"); + + final String result = authorizingFacade.getValue("param1"); + + assertEquals("value1", result); + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.READ, actionCaptor.getValue()); + } + + @Test + void testGetDefinedParameterNamesAuthorizesWithReadAction() { + when(delegate.getDefinedParameterNames()).thenReturn(Set.of("param1", "param2")); + + final Set result = authorizingFacade.getDefinedParameterNames(); + + assertEquals(Set.of("param1", "param2"), result); + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.READ, actionCaptor.getValue()); + } + + @Test + void testIsSensitiveAuthorizesWithReadAction() { + when(delegate.isSensitive("param1")).thenReturn(true); + + final boolean result = authorizingFacade.isSensitive("param1"); + + assertTrue(result); + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.READ, actionCaptor.getValue()); + } + + @Test + void testUpdateParametersAuthorizesWithWriteAction() { + authorizingFacade.updateParameters(Collections.emptyList()); + + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.WRITE, actionCaptor.getValue()); + verify(delegate).updateParameters(Collections.emptyList()); + } + + @Test + void testCreateAssetAuthorizesWithWriteAction() throws IOException { + final ByteArrayInputStream inputStream = new ByteArrayInputStream(new byte[0]); + when(delegate.createAsset(inputStream)).thenReturn(asset); + + final Asset result = authorizingFacade.createAsset(inputStream); + + assertEquals(asset, result); + final ArgumentCaptor actionCaptor = ArgumentCaptor.forClass(RequestAction.class); + verify(connectorAuthorizable).authorize(eq(authorizer), actionCaptor.capture(), any(NiFiUser.class)); + assertEquals(RequestAction.WRITE, actionCaptor.getValue()); + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/controller/ControllerFacadeTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/controller/ControllerFacadeTest.java new file mode 100644 index 000000000000..ebbdeae4ed65 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/controller/ControllerFacadeTest.java @@ -0,0 +1,531 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.controller; + +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.RequestAction; +import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserDetails; +import org.apache.nifi.authorization.user.StandardNiFiUser; +import org.apache.nifi.c2.protocol.component.api.Bundle; +import org.apache.nifi.c2.protocol.component.api.ComponentManifest; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; +import org.apache.nifi.c2.protocol.component.api.RuntimeManifest; +import org.apache.nifi.connectable.Connectable; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.manifest.RuntimeManifestService; +import org.apache.nifi.provenance.ProvenanceAuthorizableFactory; +import org.apache.nifi.provenance.ProvenanceEventRecord; +import org.apache.nifi.provenance.ProvenanceEventType; +import org.apache.nifi.provenance.ProvenanceRepository; +import org.apache.nifi.remote.RemoteGroupPort; +import org.apache.nifi.web.api.dto.provenance.ProvenanceEventDTO; +import org.apache.nifi.web.api.dto.search.SearchResultsDTO; +import org.apache.nifi.web.search.query.SearchQuery; +import org.apache.nifi.web.search.query.SearchQueryParser; +import org.apache.nifi.web.security.token.NiFiAuthenticationToken; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.security.core.context.SecurityContextHolder; + +import java.io.IOException; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +public class ControllerFacadeTest { + + private static final String SEARCH_TERM = "test-search"; + + @Mock + private SearchQueryParser searchQueryParser; + + @Mock + private ControllerSearchService controllerSearchService; + + @Mock + private ProcessGroup connectorProcessGroup; + + @Mock + private SearchQuery searchQuery; + + @Mock + private RuntimeManifestService runtimeManifestService; + + @Mock + private RuntimeManifest runtimeManifest; + + @Mock + private Bundle manifestBundle; + + @Mock + private ComponentManifest componentManifest; + + private ControllerFacade controllerFacade; + + private static final String TEST_GROUP = "org.apache.nifi"; + private static final String TEST_ARTIFACT = "nifi-test-nar"; + private static final String TEST_VERSION = "2.0.0"; + private static final String TEST_CONNECTOR_TYPE = "org.apache.nifi.connectors.TestConnector"; + + @BeforeEach + public void setUp() { + controllerFacade = new ControllerFacade(); + controllerFacade.setSearchQueryParser(searchQueryParser); + controllerFacade.setControllerSearchService(controllerSearchService); + controllerFacade.setRuntimeManifestService(runtimeManifestService); + + final NiFiUser user = new StandardNiFiUser.Builder().identity("test-user").build(); + SecurityContextHolder.getContext().setAuthentication(new NiFiAuthenticationToken(new NiFiUserDetails(user))); + } + + @Test + public void testSearchConnectorDelegatesSearchQueryParserWithConnectorProcessGroupAsRootAndActive() { + when(searchQueryParser.parse(eq(SEARCH_TERM), any(NiFiUser.class), eq(connectorProcessGroup), eq(connectorProcessGroup))).thenReturn(searchQuery); + when(searchQuery.getTerm()).thenReturn(SEARCH_TERM); + + final SearchResultsDTO results = controllerFacade.searchConnector(SEARCH_TERM, connectorProcessGroup); + + assertNotNull(results); + verify(searchQueryParser).parse(eq(SEARCH_TERM), any(NiFiUser.class), eq(connectorProcessGroup), eq(connectorProcessGroup)); + verify(controllerSearchService).search(eq(searchQuery), any(SearchResultsDTO.class)); + verify(controllerSearchService, never()).searchParameters(any(), any()); + } + + @Test + public void testSearchConnectorWithEmptyTermDoesNotInvokeSearchService() { + when(searchQueryParser.parse(eq(""), any(NiFiUser.class), eq(connectorProcessGroup), eq(connectorProcessGroup))).thenReturn(searchQuery); + when(searchQuery.getTerm()).thenReturn(""); + + final SearchResultsDTO results = controllerFacade.searchConnector("", connectorProcessGroup); + + assertNotNull(results); + verify(searchQueryParser).parse(eq(""), any(NiFiUser.class), eq(connectorProcessGroup), eq(connectorProcessGroup)); + } + + @Test + public void testSearchConnectorWithNullTermDoesNotInvokeSearchService() { + when(searchQueryParser.parse(eq(null), any(NiFiUser.class), eq(connectorProcessGroup), eq(connectorProcessGroup))).thenReturn(searchQuery); + when(searchQuery.getTerm()).thenReturn(null); + + final SearchResultsDTO results = controllerFacade.searchConnector(null, connectorProcessGroup); + + assertNotNull(results); + verify(searchQueryParser).parse(eq(null), any(NiFiUser.class), eq(connectorProcessGroup), eq(connectorProcessGroup)); + } + + @Test + public void testGetConnectorDefinitionReturnsDefinitionWhenFound() { + final ConnectorDefinition expectedDefinition = new ConnectorDefinition(); + expectedDefinition.setGroup(TEST_GROUP); + expectedDefinition.setArtifact(TEST_ARTIFACT); + expectedDefinition.setVersion(TEST_VERSION); + expectedDefinition.setType(TEST_CONNECTOR_TYPE); + + when(runtimeManifestService.getManifestForBundle(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION)).thenReturn(runtimeManifest); + when(runtimeManifest.getBundles()).thenReturn(List.of(manifestBundle)); + when(manifestBundle.getComponentManifest()).thenReturn(componentManifest); + when(componentManifest.getConnectors()).thenReturn(List.of(expectedDefinition)); + + final ConnectorDefinition result = controllerFacade.getConnectorDefinition(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION, TEST_CONNECTOR_TYPE); + + assertNotNull(result); + assertEquals(TEST_CONNECTOR_TYPE, result.getType()); + } + + @Test + public void testGetConnectorDefinitionReturnsNullWhenConnectorsListIsNull() { + when(runtimeManifestService.getManifestForBundle(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION)).thenReturn(runtimeManifest); + when(runtimeManifest.getBundles()).thenReturn(List.of(manifestBundle)); + when(manifestBundle.getComponentManifest()).thenReturn(componentManifest); + when(componentManifest.getConnectors()).thenReturn(null); + + final ConnectorDefinition result = controllerFacade.getConnectorDefinition(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION, TEST_CONNECTOR_TYPE); + + assertNull(result); + } + + @Test + public void testGetConnectorDefinitionReturnsNullWhenConnectorTypeNotFound() { + final ConnectorDefinition otherDefinition = new ConnectorDefinition(); + otherDefinition.setType("org.apache.nifi.connectors.OtherConnector"); + + when(runtimeManifestService.getManifestForBundle(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION)).thenReturn(runtimeManifest); + when(runtimeManifest.getBundles()).thenReturn(List.of(manifestBundle)); + when(manifestBundle.getComponentManifest()).thenReturn(componentManifest); + when(componentManifest.getConnectors()).thenReturn(List.of(otherDefinition)); + + final ConnectorDefinition result = controllerFacade.getConnectorDefinition(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION, TEST_CONNECTOR_TYPE); + + assertNull(result); + } + + @Test + public void testGetConnectorDefinitionReturnsNullWhenConnectorsListIsEmpty() { + when(runtimeManifestService.getManifestForBundle(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION)).thenReturn(runtimeManifest); + when(runtimeManifest.getBundles()).thenReturn(List.of(manifestBundle)); + when(manifestBundle.getComponentManifest()).thenReturn(componentManifest); + when(componentManifest.getConnectors()).thenReturn(Collections.emptyList()); + + final ConnectorDefinition result = controllerFacade.getConnectorDefinition(TEST_GROUP, TEST_ARTIFACT, TEST_VERSION, TEST_CONNECTOR_TYPE); + + assertNull(result); + } + + @Test + public void testGetProvenanceEventSetsConnectorIdWhenComponentBelongsToConnector() throws IOException { + final String componentId = "test-component-id"; + final String groupId = "test-group-id"; + final String connectorId = "test-connector-id"; + + final ControllerFacade facade = createProvenanceFacade(componentId, groupId, Optional.of(connectorId)); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertEquals(connectorId, result.getConnectorId()); + assertEquals(groupId, result.getGroupId()); + assertEquals(componentId, result.getComponentId()); + } + + @Test + public void testGetProvenanceEventConnectorIdIsNullWhenComponentNotManagedByConnector() throws IOException { + final String componentId = "test-component-id"; + final String groupId = "test-group-id"; + + final ControllerFacade facade = createProvenanceFacade(componentId, groupId, Optional.empty()); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertNull(result.getConnectorId()); + assertEquals(groupId, result.getGroupId()); + assertEquals(componentId, result.getComponentId()); + } + + @Test + public void testGetProvenanceEventSetsDetailsForConnectionInRootGroup() throws IOException { + final String connectionId = "root-connection-id"; + final String groupId = "connection-group-id"; + final String connectorId = "connector-id"; + final String connectionName = "My Connection"; + + final ControllerFacade facade = createConnectionProvenanceFacade(connectionId, groupId, + Optional.of(connectorId), connectionName); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertEquals(groupId, result.getGroupId()); + assertEquals(connectorId, result.getConnectorId()); + assertEquals(connectionName, result.getComponentName()); + } + + @Test + public void testGetProvenanceEventSetsDetailsForConnectionInConnectorManagedGroup() throws IOException { + final String connectionId = "connector-managed-connection-id"; + final String groupId = "managed-group-id"; + final String connectorId = "managed-connector-id"; + final String connectionName = "Managed Connection"; + + final ControllerFacade facade = createConnectionProvenanceFacade(connectionId, groupId, + Optional.of(connectorId), connectionName); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertEquals(groupId, result.getGroupId()); + assertEquals(connectorId, result.getConnectorId()); + assertEquals(connectionName, result.getComponentName()); + } + + @Test + public void testGetProvenanceEventSetsDetailsForConnectionWithNoConnectorId() throws IOException { + final String connectionId = "plain-connection-id"; + final String groupId = "plain-group-id"; + + final ControllerFacade facade = createConnectionProvenanceFacade(connectionId, groupId, + Optional.empty(), "Plain Connection"); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertEquals(groupId, result.getGroupId()); + assertNull(result.getConnectorId()); + } + + @Test + public void testGetProvenanceEventSetsDetailsForRemoteGroupPort() throws IOException { + final String remotePortId = "remote-port-id"; + final String groupId = "remote-port-group-id"; + final String connectorId = "remote-connector-id"; + final String portName = "Remote Port"; + + final ControllerFacade facade = createRemoteGroupPortProvenanceFacade(remotePortId, groupId, + Optional.of(connectorId), portName); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertEquals(groupId, result.getGroupId()); + assertEquals(connectorId, result.getConnectorId()); + assertEquals(portName, result.getComponentName()); + } + + @Test + public void testGetProvenanceEventSetsDetailsForRemoteGroupPortInConnectorManagedGroup() throws IOException { + final String remotePortId = "managed-remote-port-id"; + final String groupId = "managed-remote-group-id"; + final String connectorId = "managed-remote-connector-id"; + final String portName = "Managed Remote Port"; + + final ControllerFacade facade = createRemoteGroupPortProvenanceFacade(remotePortId, groupId, + Optional.of(connectorId), portName); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertEquals(groupId, result.getGroupId()); + assertEquals(connectorId, result.getConnectorId()); + assertEquals(portName, result.getComponentName()); + } + + @Test + public void testGetProvenanceEventComponentNotFoundLeavesFieldsUnset() throws IOException { + final String componentId = "unknown-component-id"; + + final ControllerFacade facade = createNotFoundProvenanceFacade(componentId); + + final ProvenanceEventDTO result = facade.getProvenanceEvent(1L); + + assertNotNull(result); + assertNull(result.getGroupId()); + assertNull(result.getConnectorId()); + assertNull(result.getComponentName()); + } + + /** + * Creates a ControllerFacade wired with mocks sufficient to test provenance event creation + * through the connectable branch of setComponentDetails. + */ + private ControllerFacade createProvenanceFacade(final String componentId, final String groupId, + final Optional connectorIdentifier) throws IOException { + // Mock the ProvenanceEventRecord + final ProvenanceEventRecord event = mock(ProvenanceEventRecord.class); + when(event.getEventId()).thenReturn(1L); + when(event.getEventTime()).thenReturn(System.currentTimeMillis()); + when(event.getEventType()).thenReturn(ProvenanceEventType.CREATE); + when(event.getFlowFileUuid()).thenReturn("test-flowfile-uuid"); + when(event.getFileSize()).thenReturn(1024L); + when(event.getComponentId()).thenReturn(componentId); + when(event.getComponentType()).thenReturn("TestProcessor"); + when(event.getUpdatedAttributes()).thenReturn(Map.of()); + when(event.getPreviousAttributes()).thenReturn(Map.of()); + when(event.getParentUuids()).thenReturn(List.of()); + when(event.getChildUuids()).thenReturn(List.of()); + when(event.getEventDuration()).thenReturn(-1L); + when(event.getLineageStartDate()).thenReturn(0L); + + // Mock the ProcessGroup that the connectable belongs to + final ProcessGroup processGroup = mock(ProcessGroup.class); + when(processGroup.getIdentifier()).thenReturn(groupId); + when(processGroup.getConnectorIdentifier()).thenReturn(connectorIdentifier); + + // Mock a Connectable found in the flow + final Connectable connectable = mock(Connectable.class); + when(connectable.getProcessGroup()).thenReturn(processGroup); + when(connectable.getName()).thenReturn("Test Component"); + when(connectable.checkAuthorization(any(), any(), any())).thenReturn(AuthorizationResult.approved()); + + // Mock FlowManager + final FlowManager flowManager = mock(FlowManager.class); + when(flowManager.findConnectable(componentId)).thenReturn(connectable); + + // Mock ProvenanceRepository + final ProvenanceRepository provenanceRepository = mock(ProvenanceRepository.class); + when(provenanceRepository.getEvent(eq(1L), any(NiFiUser.class))).thenReturn(event); + + // Mock data authorization as not approved so we skip content availability logic + final Authorizable dataAuthorizable = mock(Authorizable.class); + when(dataAuthorizable.checkAuthorization(any(), eq(RequestAction.READ), any(NiFiUser.class), any())) + .thenReturn(AuthorizationResult.denied("test")); + + // Mock ProvenanceAuthorizableFactory + final ProvenanceAuthorizableFactory provenanceAuthorizableFactory = mock(ProvenanceAuthorizableFactory.class); + when(provenanceAuthorizableFactory.createLocalDataAuthorizable(componentId)).thenReturn(dataAuthorizable); + + // Mock FlowController + final FlowController flowController = mock(FlowController.class); + when(flowController.getFlowManager()).thenReturn(flowManager); + when(flowController.getProvenanceRepository()).thenReturn(provenanceRepository); + when(flowController.getProvenanceAuthorizableFactory()).thenReturn(provenanceAuthorizableFactory); + + // Mock Authorizer + final Authorizer authorizer = mock(Authorizer.class); + + // Build the ControllerFacade + final ControllerFacade facade = new ControllerFacade(); + facade.setFlowController(flowController); + facade.setAuthorizer(authorizer); + + return facade; + } + + /** + * Creates a ControllerFacade wired with mocks to test provenance event creation + * through the connection branch of setComponentDetails. + */ + private ControllerFacade createConnectionProvenanceFacade(final String connectionId, final String groupId, + final Optional connectorIdentifier, + final String connectionName) throws IOException { + final ProvenanceEventRecord event = createMockProvenanceEvent(connectionId); + + final ProcessGroup connectionGroup = mock(ProcessGroup.class); + when(connectionGroup.getIdentifier()).thenReturn(groupId); + when(connectionGroup.getConnectorIdentifier()).thenReturn(connectorIdentifier); + + final Connection connection = mock(Connection.class); + when(connection.getProcessGroup()).thenReturn(connectionGroup); + when(connection.getName()).thenReturn(connectionName); + when(connection.getRelationships()).thenReturn(List.of()); + when(connection.checkAuthorization(any(), any(), any())).thenReturn(AuthorizationResult.approved()); + + final FlowManager flowManager = mock(FlowManager.class); + when(flowManager.findConnectable(connectionId)).thenReturn(null); + + final FlowController flowController = createMockFlowController(connectionId, event, flowManager); + when(flowController.findRemoteGroupPortIncludingConnectorManaged(connectionId)).thenReturn(null); + when(flowController.findConnectionIncludingConnectorManaged(connectionId)).thenReturn(connection); + + final ControllerFacade facade = new ControllerFacade(); + facade.setFlowController(flowController); + facade.setAuthorizer(mock(Authorizer.class)); + + return facade; + } + + /** + * Creates a ControllerFacade wired with mocks to test provenance event creation + * through the remote group port branch of setComponentDetails. + */ + private ControllerFacade createRemoteGroupPortProvenanceFacade(final String remotePortId, final String groupId, + final Optional connectorIdentifier, + final String portName) throws IOException { + final ProvenanceEventRecord event = createMockProvenanceEvent(remotePortId); + + final ProcessGroup portGroup = mock(ProcessGroup.class); + when(portGroup.getIdentifier()).thenReturn(groupId); + when(portGroup.getConnectorIdentifier()).thenReturn(connectorIdentifier); + + final RemoteGroupPort remoteGroupPort = mock(RemoteGroupPort.class); + when(remoteGroupPort.getProcessGroup()).thenReturn(portGroup); + when(remoteGroupPort.getName()).thenReturn(portName); + when(remoteGroupPort.checkAuthorization(any(), any(), any())).thenReturn(AuthorizationResult.approved()); + + final FlowManager flowManager = mock(FlowManager.class); + when(flowManager.findConnectable(remotePortId)).thenReturn(null); + + final FlowController flowController = createMockFlowController(remotePortId, event, flowManager); + when(flowController.findRemoteGroupPortIncludingConnectorManaged(remotePortId)).thenReturn(remoteGroupPort); + + final ControllerFacade facade = new ControllerFacade(); + facade.setFlowController(flowController); + facade.setAuthorizer(mock(Authorizer.class)); + + return facade; + } + + /** + * Creates a ControllerFacade wired with mocks where the component is not found + * through any branch of setComponentDetails. + */ + private ControllerFacade createNotFoundProvenanceFacade(final String componentId) throws IOException { + final ProvenanceEventRecord event = createMockProvenanceEvent(componentId); + + final FlowManager flowManager = mock(FlowManager.class); + when(flowManager.findConnectable(componentId)).thenReturn(null); + + final FlowController flowController = createMockFlowController(componentId, event, flowManager); + when(flowController.findRemoteGroupPortIncludingConnectorManaged(componentId)).thenReturn(null); + when(flowController.findConnectionIncludingConnectorManaged(componentId)).thenReturn(null); + + final ControllerFacade facade = new ControllerFacade(); + facade.setFlowController(flowController); + facade.setAuthorizer(mock(Authorizer.class)); + + return facade; + } + + private ProvenanceEventRecord createMockProvenanceEvent(final String componentId) { + final ProvenanceEventRecord event = mock(ProvenanceEventRecord.class); + when(event.getEventId()).thenReturn(1L); + when(event.getEventTime()).thenReturn(System.currentTimeMillis()); + when(event.getEventType()).thenReturn(ProvenanceEventType.DROP); + when(event.getFlowFileUuid()).thenReturn("test-flowfile-uuid"); + when(event.getFileSize()).thenReturn(1024L); + when(event.getComponentId()).thenReturn(componentId); + when(event.getComponentType()).thenReturn("Connection"); + when(event.getUpdatedAttributes()).thenReturn(Map.of()); + when(event.getPreviousAttributes()).thenReturn(Map.of()); + when(event.getParentUuids()).thenReturn(List.of()); + when(event.getChildUuids()).thenReturn(List.of()); + when(event.getEventDuration()).thenReturn(-1L); + when(event.getLineageStartDate()).thenReturn(0L); + return event; + } + + private FlowController createMockFlowController(final String componentId, final ProvenanceEventRecord event, + final FlowManager flowManager) throws IOException { + final ProvenanceRepository provenanceRepository = mock(ProvenanceRepository.class); + when(provenanceRepository.getEvent(eq(1L), any(NiFiUser.class))).thenReturn(event); + + final Authorizable dataAuthorizable = mock(Authorizable.class); + when(dataAuthorizable.checkAuthorization(any(), eq(RequestAction.READ), any(NiFiUser.class), any())) + .thenReturn(AuthorizationResult.denied("test")); + + final ProvenanceAuthorizableFactory provenanceAuthorizableFactory = mock(ProvenanceAuthorizableFactory.class); + when(provenanceAuthorizableFactory.createLocalDataAuthorizable(componentId)).thenReturn(dataAuthorizable); + + final FlowController flowController = mock(FlowController.class); + when(flowController.getFlowManager()).thenReturn(flowManager); + when(flowController.getProvenanceRepository()).thenReturn(provenanceRepository); + when(flowController.getProvenanceAuthorizableFactory()).thenReturn(provenanceAuthorizableFactory); + + return flowController; + } +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardConnectionDAOTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardConnectionDAOTest.java new file mode 100644 index 000000000000..7aa6dc19d224 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardConnectionDAOTest.java @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.dao.impl; + +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.connectable.Connection; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.web.ResourceNotFoundException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class StandardConnectionDAOTest { + + private StandardConnectionDAO connectionDAO; + + @Mock + private FlowController flowController; + + @Mock + private FlowManager flowManager; + + @Mock + private ProcessGroup rootGroup; + + @Mock + private ConnectorRepository connectorRepository; + + @Mock + private ConnectorNode connectorNode; + + @Mock + private FrameworkFlowContext frameworkFlowContext; + + @Mock + private ProcessGroup connectorManagedGroup; + + @Mock + private Connection rootConnection; + + @Mock + private Connection connectorConnection; + + private static final String ROOT_CONNECTION_ID = "root-connection-id"; + private static final String CONNECTOR_CONNECTION_ID = "connector-connection-id"; + private static final String NON_EXISTENT_ID = "non-existent-id"; + + @BeforeEach + void setUp() { + connectionDAO = new StandardConnectionDAO(); + connectionDAO.setFlowController(flowController); + + when(flowController.getFlowManager()).thenReturn(flowManager); + when(flowManager.getRootGroup()).thenReturn(rootGroup); + when(flowController.getConnectorRepository()).thenReturn(connectorRepository); + + // Setup root group connection + when(rootGroup.findConnection(ROOT_CONNECTION_ID)).thenReturn(rootConnection); + when(rootGroup.findConnection(CONNECTOR_CONNECTION_ID)).thenReturn(null); + when(rootGroup.findConnection(NON_EXISTENT_ID)).thenReturn(null); + + // Setup connector managed group + when(connectorRepository.getConnectors()).thenReturn(List.of(connectorNode)); + when(connectorNode.getActiveFlowContext()).thenReturn(frameworkFlowContext); + when(frameworkFlowContext.getManagedProcessGroup()).thenReturn(connectorManagedGroup); + when(connectorManagedGroup.findConnection(CONNECTOR_CONNECTION_ID)).thenReturn(connectorConnection); + when(connectorManagedGroup.findConnection(ROOT_CONNECTION_ID)).thenReturn(null); + when(connectorManagedGroup.findConnection(NON_EXISTENT_ID)).thenReturn(null); + } + + @Test + void testGetConnectionFromRootGroup() { + final Connection result = connectionDAO.getConnection(ROOT_CONNECTION_ID); + + assertEquals(rootConnection, result); + } + + @Test + void testGetConnectionFromRootGroupWithIncludeConnectorManagedFalse() { + final Connection result = connectionDAO.getConnection(ROOT_CONNECTION_ID, false); + + assertEquals(rootConnection, result); + } + + @Test + void testGetConnectionFromRootGroupWithIncludeConnectorManagedTrue() { + final Connection result = connectionDAO.getConnection(ROOT_CONNECTION_ID, true); + + assertEquals(rootConnection, result); + } + + @Test + void testGetConnectionFromConnectorManagedGroupWithIncludeConnectorManagedTrue() { + final Connection result = connectionDAO.getConnection(CONNECTOR_CONNECTION_ID, true); + + assertEquals(connectorConnection, result); + } + + @Test + void testGetConnectionFromConnectorManagedGroupWithIncludeConnectorManagedFalseThrows() { + assertThrows(ResourceNotFoundException.class, () -> + connectionDAO.getConnection(CONNECTOR_CONNECTION_ID, false) + ); + } + + @Test + void testGetConnectionWithDefaultDoesNotFindConnectorManagedConnection() { + assertThrows(ResourceNotFoundException.class, () -> + connectionDAO.getConnection(CONNECTOR_CONNECTION_ID) + ); + } + + @Test + void testGetConnectionWithNonExistentIdThrows() { + assertThrows(ResourceNotFoundException.class, () -> + connectionDAO.getConnection(NON_EXISTENT_ID, true) + ); + } + + @Test + void testHasConnectionInRootGroup() { + assertTrue(connectionDAO.hasConnection(ROOT_CONNECTION_ID)); + } + + @Test + void testHasConnectionNotInRootGroup() { + // hasConnection only checks the root group, not connector-managed groups + assertFalse(connectionDAO.hasConnection(CONNECTOR_CONNECTION_ID)); + } + + @Test + void testHasConnectionWithNonExistentId() { + assertFalse(connectionDAO.hasConnection(NON_EXISTENT_ID)); + } + + @Test + void testGetConnectionFromConnectorWithNullActiveFlowContext() { + // Simulate connector with no active flow context + when(connectorNode.getActiveFlowContext()).thenReturn(null); + + assertThrows(ResourceNotFoundException.class, () -> + connectionDAO.getConnection(CONNECTOR_CONNECTION_ID, true) + ); + } + + @Test + void testGetConnectionWithMultipleConnectors() { + // Setup a second connector + final ConnectorNode connectorNode2 = org.mockito.Mockito.mock(ConnectorNode.class); + final FrameworkFlowContext flowContext2 = org.mockito.Mockito.mock(FrameworkFlowContext.class); + final ProcessGroup managedGroup2 = org.mockito.Mockito.mock(ProcessGroup.class); + final Connection connectionInSecondConnector = org.mockito.Mockito.mock(Connection.class); + final String secondConnectorConnectionId = "second-connector-connection-id"; + + when(connectorRepository.getConnectors()).thenReturn(List.of(connectorNode, connectorNode2)); + when(connectorNode2.getActiveFlowContext()).thenReturn(flowContext2); + when(flowContext2.getManagedProcessGroup()).thenReturn(managedGroup2); + when(managedGroup2.findConnection(secondConnectorConnectionId)).thenReturn(connectionInSecondConnector); + + final Connection result = connectionDAO.getConnection(secondConnectorConnectionId, true); + + assertEquals(connectionInSecondConnector, result); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardConnectorDAOTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardConnectorDAOTest.java new file mode 100644 index 000000000000..d1c589fd2296 --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardConnectorDAOTest.java @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.dao.impl; + +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.DescribedValue; +import org.apache.nifi.components.connector.ConnectorConfiguration; +import org.apache.nifi.components.connector.ConnectorNode; +import org.apache.nifi.components.connector.ConnectorRepository; +import org.apache.nifi.components.connector.ConnectorUpdateContext; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.FrameworkFlowContext; +import org.apache.nifi.components.connector.MutableConnectorConfigurationContext; +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.web.NiFiCoreException; +import org.apache.nifi.web.ResourceNotFoundException; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InOrder; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class StandardConnectorDAOTest { + + private StandardConnectorDAO connectorDAO; + + @Mock + private FlowController flowController; + + @Mock + private ConnectorRepository connectorRepository; + + @Mock + private ConnectorNode connectorNode; + + @Mock + private ConnectorUpdateContext connectorUpdateContext; + + @Mock + private FrameworkFlowContext frameworkFlowContext; + + @Mock + private MutableConnectorConfigurationContext configurationContext; + + @Mock + private ConnectorConfiguration connectorConfiguration; + + private static final String CONNECTOR_ID = "test-connector-id"; + private static final String STEP_NAME = "test-step"; + private static final String PROPERTY_NAME = "test-property"; + + @BeforeEach + void setUp() { + connectorDAO = new StandardConnectorDAO(); + connectorDAO.setFlowController(flowController); + + when(flowController.getConnectorRepository()).thenReturn(connectorRepository); + + final MutableConnectorConfigurationContext configContext = mock(MutableConnectorConfigurationContext.class); + when(configContext.toConnectorConfiguration()).thenReturn(mock(ConnectorConfiguration.class)); + final FrameworkFlowContext activeContext = mock(FrameworkFlowContext.class); + when(activeContext.getConfigurationContext()).thenReturn(configContext); + when(connectorNode.getActiveFlowContext()).thenReturn(activeContext); + } + + @Test + void testApplyConnectorUpdate() throws Exception { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + + connectorDAO.applyConnectorUpdate(CONNECTOR_ID, connectorUpdateContext); + + verify(connectorRepository).getConnector(CONNECTOR_ID); + verify(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + } + + @Test + void testApplyConnectorUpdateWithNonExistentConnector() throws Exception { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(null); + + final ResourceNotFoundException exception = assertThrows(ResourceNotFoundException.class, () -> + connectorDAO.applyConnectorUpdate(CONNECTOR_ID, connectorUpdateContext) + ); + + assertEquals("Could not find Connector with ID " + CONNECTOR_ID, exception.getMessage()); + verify(connectorRepository).getConnector(CONNECTOR_ID); + verify(connectorRepository, never()).applyUpdate(any(ConnectorNode.class), any(ConnectorUpdateContext.class)); + } + + @Test + void testApplyConnectorUpdateWithFlowUpdateException() throws Exception { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + doThrow(new FlowUpdateException("Flow update failed")).when(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + + final NiFiCoreException exception = assertThrows(NiFiCoreException.class, () -> + connectorDAO.applyConnectorUpdate(CONNECTOR_ID, connectorUpdateContext) + ); + + assertEquals("Failed to apply connector update: org.apache.nifi.components.connector.FlowUpdateException: Flow update failed", exception.getMessage()); + verify(connectorRepository).getConnector(CONNECTOR_ID); + verify(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + } + + @Test + void testApplyConnectorUpdateWithRuntimeException() throws Exception { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + doThrow(new RuntimeException("Test exception")).when(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + + final NiFiCoreException exception = assertThrows(NiFiCoreException.class, () -> + connectorDAO.applyConnectorUpdate(CONNECTOR_ID, connectorUpdateContext) + ); + + assertEquals("Failed to apply connector update: java.lang.RuntimeException: Test exception", exception.getMessage()); + verify(connectorRepository).getConnector(CONNECTOR_ID); + verify(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + } + + @Test + void testApplyConnectorUpdateWithNullException() throws Exception { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + doThrow(new RuntimeException()).when(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + + final NiFiCoreException exception = assertThrows(NiFiCoreException.class, () -> + connectorDAO.applyConnectorUpdate(CONNECTOR_ID, connectorUpdateContext) + ); + + assertEquals("Failed to apply connector update: java.lang.RuntimeException", exception.getMessage()); + verify(connectorRepository).getConnector(CONNECTOR_ID); + verify(connectorRepository).applyUpdate(connectorNode, connectorUpdateContext); + } + + @Test + void testGetConnectorWithNonExistentId() { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(null); + + assertThrows(ResourceNotFoundException.class, () -> + connectorDAO.getConnector(CONNECTOR_ID) + ); + + verify(connectorRepository).getConnector(CONNECTOR_ID); + } + + @Test + void testGetConnectorSuccess() { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + + final ConnectorNode result = connectorDAO.getConnector(CONNECTOR_ID); + + assertEquals(connectorNode, result); + verify(connectorRepository).getConnector(CONNECTOR_ID); + } + + @Test + void testFetchAllowableValuesWithoutFilter() { + final List expectedValues = List.of( + new AllowableValue("value1", "Value 1", "First value"), + new AllowableValue("value2", "Value 2", "Second value") + ); + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + when(connectorNode.fetchAllowableValues(STEP_NAME, PROPERTY_NAME)).thenReturn(expectedValues); + + final List result = connectorDAO.fetchAllowableValues(CONNECTOR_ID, STEP_NAME, PROPERTY_NAME, null); + + assertEquals(expectedValues, result); + verify(connectorNode).fetchAllowableValues(STEP_NAME, PROPERTY_NAME); + verify(connectorNode, never()).fetchAllowableValues(any(), any(), any()); + } + + @Test + void testFetchAllowableValuesWithEmptyFilter() { + final List expectedValues = List.of( + new AllowableValue("value1", "Value 1", "First value") + ); + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + when(connectorNode.fetchAllowableValues(STEP_NAME, PROPERTY_NAME)).thenReturn(expectedValues); + + final List result = connectorDAO.fetchAllowableValues(CONNECTOR_ID, STEP_NAME, PROPERTY_NAME, ""); + + assertEquals(expectedValues, result); + verify(connectorNode).fetchAllowableValues(STEP_NAME, PROPERTY_NAME); + verify(connectorNode, never()).fetchAllowableValues(any(), any(), any()); + } + + @Test + void testFetchAllowableValuesWithFilter() { + final String filter = "test-filter"; + final List expectedValues = List.of( + new AllowableValue("filtered-value", "Filtered Value", "Filtered result") + ); + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + when(connectorNode.fetchAllowableValues(STEP_NAME, PROPERTY_NAME, filter)).thenReturn(expectedValues); + + final List result = connectorDAO.fetchAllowableValues(CONNECTOR_ID, STEP_NAME, PROPERTY_NAME, filter); + + assertEquals(expectedValues, result); + verify(connectorNode).fetchAllowableValues(STEP_NAME, PROPERTY_NAME, filter); + verify(connectorNode, never()).fetchAllowableValues(STEP_NAME, PROPERTY_NAME); + } + + @Test + void testFetchAllowableValuesWithNonExistentConnector() { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(null); + + assertThrows(ResourceNotFoundException.class, () -> + connectorDAO.fetchAllowableValues(CONNECTOR_ID, STEP_NAME, PROPERTY_NAME, null) + ); + + verify(connectorRepository).getConnector(CONNECTOR_ID); + } + + @Test + void testVerifyConfigurationStepSyncsAssetsBeforeVerification() { + when(connectorRepository.getConnector(CONNECTOR_ID)).thenReturn(connectorNode); + final ConfigurationStepConfigurationDTO stepConfigDto = new ConfigurationStepConfigurationDTO(); + + connectorDAO.verifyConfigurationStep(CONNECTOR_ID, STEP_NAME, stepConfigDto); + + final InOrder inOrder = inOrder(connectorRepository, connectorNode); + inOrder.verify(connectorRepository).syncAssetsFromProvider(connectorNode); + inOrder.verify(connectorNode).verifyConfigurationStep(any(), any()); + } + + @Test + void testDeleteConnectorRemovesAssetsAndConnector() { + connectorDAO.deleteConnector(CONNECTOR_ID); + + verify(connectorRepository).deleteAssets(CONNECTOR_ID); + verify(connectorRepository).removeConnector(CONNECTOR_ID); + } + + @Test + void testDeleteConnectorDoesNotRemoveConnectorWhenAssetDeletionFails() { + doThrow(new RuntimeException("Asset deletion failed")).when(connectorRepository).deleteAssets(CONNECTOR_ID); + + assertThrows(RuntimeException.class, () -> + connectorDAO.deleteConnector(CONNECTOR_ID) + ); + + verify(connectorRepository).deleteAssets(CONNECTOR_ID); + verify(connectorRepository, never()).removeConnector(any()); + } + + @Test + void testVerifyCreateDelegatesToRepository() { + final ConnectorDTO connectorDTO = new ConnectorDTO(); + connectorDTO.setId(CONNECTOR_ID); + + connectorDAO.verifyCreate(connectorDTO); + + verify(connectorRepository).verifyCreate(CONNECTOR_ID); + } + + @Test + void testVerifyCreateWithRepositoryThrowingException() { + final ConnectorDTO connectorDTO = new ConnectorDTO(); + connectorDTO.setId(CONNECTOR_ID); + + doThrow(new IllegalStateException("A Connector already exists with ID %s".formatted(CONNECTOR_ID))) + .when(connectorRepository).verifyCreate(CONNECTOR_ID); + + final IllegalStateException exception = assertThrows(IllegalStateException.class, () -> + connectorDAO.verifyCreate(connectorDTO) + ); + + assertEquals("A Connector already exists with ID %s".formatted(CONNECTOR_ID), exception.getMessage()); + } + + @Test + void testVerifyCreateWithNullId() { + final ConnectorDTO connectorDTO = new ConnectorDTO(); + connectorDTO.setId(null); + + connectorDAO.verifyCreate(connectorDTO); + + verify(connectorRepository, never()).verifyCreate(any()); + } + +} + diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAOTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAOTest.java new file mode 100644 index 000000000000..cf5441f199ab --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessGroupDAOTest.java @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.dao.impl; + +import org.apache.nifi.controller.FlowController; +import org.apache.nifi.controller.flow.FlowManager; +import org.apache.nifi.groups.ProcessGroup; +import org.apache.nifi.web.ResourceNotFoundException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class StandardProcessGroupDAOTest { + + private StandardProcessGroupDAO processGroupDAO; + + @Mock + private FlowController flowController; + + @Mock + private FlowManager flowManager; + + @Mock + private ProcessGroup rootGroup; + + @Mock + private ProcessGroup connectorManagedGroup; + + private static final String ROOT_GROUP_ID = "root-group-id"; + private static final String CONNECTOR_GROUP_ID = "connector-group-id"; + private static final String NON_EXISTENT_ID = "non-existent-id"; + + @BeforeEach + void setUp() { + processGroupDAO = new StandardProcessGroupDAO(); + processGroupDAO.setFlowController(flowController); + + when(flowController.getFlowManager()).thenReturn(flowManager); + + // Setup root group lookup (non-connector managed) + when(flowManager.getGroup(ROOT_GROUP_ID, null)).thenReturn(rootGroup); + when(flowManager.getGroup(CONNECTOR_GROUP_ID, null)).thenReturn(null); + when(flowManager.getGroup(NON_EXISTENT_ID, null)).thenReturn(null); + + // Setup connector-managed group lookup (includes all groups) + when(flowManager.getGroup(ROOT_GROUP_ID)).thenReturn(rootGroup); + when(flowManager.getGroup(CONNECTOR_GROUP_ID)).thenReturn(connectorManagedGroup); + when(flowManager.getGroup(NON_EXISTENT_ID)).thenReturn(null); + } + + @Test + void testGetProcessGroupFromRootHierarchy() { + final ProcessGroup result = processGroupDAO.getProcessGroup(ROOT_GROUP_ID); + + assertEquals(rootGroup, result); + } + + @Test + void testGetProcessGroupFromRootHierarchyWithIncludeConnectorManagedFalse() { + final ProcessGroup result = processGroupDAO.getProcessGroup(ROOT_GROUP_ID, false); + + assertEquals(rootGroup, result); + } + + @Test + void testGetProcessGroupFromRootHierarchyWithIncludeConnectorManagedTrue() { + final ProcessGroup result = processGroupDAO.getProcessGroup(ROOT_GROUP_ID, true); + + assertEquals(rootGroup, result); + } + + @Test + void testGetProcessGroupFromConnectorManagedWithIncludeConnectorManagedTrue() { + final ProcessGroup result = processGroupDAO.getProcessGroup(CONNECTOR_GROUP_ID, true); + + assertEquals(connectorManagedGroup, result); + } + + @Test + void testGetProcessGroupFromConnectorManagedWithIncludeConnectorManagedFalseThrows() { + assertThrows(ResourceNotFoundException.class, () -> + processGroupDAO.getProcessGroup(CONNECTOR_GROUP_ID, false) + ); + } + + @Test + void testGetProcessGroupWithDefaultDoesNotFindConnectorManagedGroup() { + assertThrows(ResourceNotFoundException.class, () -> + processGroupDAO.getProcessGroup(CONNECTOR_GROUP_ID) + ); + } + + @Test + void testGetProcessGroupWithNonExistentIdThrows() { + assertThrows(ResourceNotFoundException.class, () -> + processGroupDAO.getProcessGroup(NON_EXISTENT_ID, true) + ); + } + + @Test + void testGetProcessGroupWithNonExistentIdAndIncludeConnectorManagedFalseThrows() { + assertThrows(ResourceNotFoundException.class, () -> + processGroupDAO.getProcessGroup(NON_EXISTENT_ID, false) + ); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessorDAOTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessorDAOTest.java index 30c9fdfdc979..1ae61d3dce6c 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessorDAOTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/StandardProcessorDAOTest.java @@ -210,7 +210,7 @@ void testCreateProcessor(@TempDir final File tempDir) { final Bundle bundle = new Bundle(bundleDetails, getClass().getClassLoader()); final List bundles = List.of(bundle); - when(flowManager.getGroup(eq(groupId))).thenReturn(processGroup); + when(flowManager.getGroup(eq(groupId), eq(null))).thenReturn(processGroup); when(flowController.getExtensionManager()).thenReturn(extensionManager); when(flowManager.createProcessor(eq(processorType), eq(id), eq(bundleCoordinate))).thenReturn(processorNode); when(extensionManager.getBundles(eq(processorType))).thenReturn(bundles); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardParameterContextDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardParameterContextDAO.java index bbba23b38bfa..60531f909d70 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardParameterContextDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardParameterContextDAO.java @@ -84,7 +84,7 @@ void setUp() { final NiFiUserDetails userDetail = new NiFiUserDetails(user); when(authentication.getPrincipal()).thenReturn(userDetail); - final ParameterReferenceManager parameterReferenceManager = new StandardParameterReferenceManager(flowController.getFlowManager()); + final ParameterReferenceManager parameterReferenceManager = new StandardParameterReferenceManager(() -> flowController.getFlowManager().getRootGroup()); final FlowManager flowManager = flowController.getFlowManager(); final StandardParameterContextManager parameterContextLookup = new StandardParameterContextManager(); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardProcessGroupDAO.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardProcessGroupDAO.java index 9ca3f0a837f0..85c55f88f9f7 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardProcessGroupDAO.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/dao/impl/TestStandardProcessGroupDAO.java @@ -59,21 +59,12 @@ void setUp() { testSubject = new StandardProcessGroupDAO(); testSubject.setFlowController(flowController); - when(flowController - .getFlowManager() - .getGroup(PARENT_PROCESS_GROUP_ID) - ).thenReturn(parentProcessGroup); - - when(flowController - .getFlowManager() - .getParameterContextManager() - .getParameterContext(PARAMETER_CONTEXT_ID) - ).thenReturn(parameterContext); + when(flowController.getFlowManager().getGroup(PARENT_PROCESS_GROUP_ID, null)).thenReturn(parentProcessGroup); + when(flowController.getFlowManager().getParameterContextManager().getParameterContext(PARAMETER_CONTEXT_ID)).thenReturn(parameterContext); } @Test public void testCreateProcessGroup() { - //GIVEN ParameterContextReferenceEntity parameterContextReferenceEntity = new ParameterContextReferenceEntity(); parameterContextReferenceEntity.setId(PARAMETER_CONTEXT_ID); @@ -84,10 +75,8 @@ public void testCreateProcessGroup() { processGroupDTO.setPosition(new PositionDTO(10.0, 20.0)); processGroupDTO.setParameterContext(parameterContextReferenceEntity); - //WHEN ProcessGroup createdProcessGroup = testSubject.createProcessGroup(PARENT_PROCESS_GROUP_ID, processGroupDTO); - //THEN verify(createdProcessGroup).setParent(parentProcessGroup); verify(createdProcessGroup).setParameterContext(parameterContext); verify(createdProcessGroup).setName(PROCESS_GROUP_NAME); diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/filter/ConnectorRequestContextFilterTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/filter/ConnectorRequestContextFilterTest.java new file mode 100644 index 000000000000..1ceef675efbf --- /dev/null +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/filter/ConnectorRequestContextFilterTest.java @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.filter; + +import jakarta.servlet.FilterChain; +import jakarta.servlet.ServletResponse; +import org.apache.nifi.authorization.user.NiFiUser; +import org.apache.nifi.authorization.user.NiFiUserDetails; +import org.apache.nifi.components.connector.ConnectorRequestContext; +import org.apache.nifi.components.connector.ConnectorRequestContextHolder; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.quality.Strictness; +import org.springframework.mock.web.MockHttpServletRequest; +import org.springframework.security.authentication.TestingAuthenticationToken; +import org.springframework.security.core.context.SecurityContextHolder; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; +import static org.mockito.Mockito.withSettings; + +@ExtendWith(MockitoExtension.class) +class ConnectorRequestContextFilterTest { + + private static final String TEST_USER = "test.user@apache.org"; + private static final String TOKEN_HEADER = "API-Key"; + private static final String TOKEN_VALUE = "test-api-key-123"; + private static final String ROLE_HEADER = "API-Role"; + private static final String ROLE_VALUE = "SERVICE_ROLE"; + + @Mock + private FilterChain filterChain; + + @Mock + private ServletResponse response; + + @AfterEach + void cleanup() { + ConnectorRequestContextHolder.clearContext(); + SecurityContextHolder.clearContext(); + } + + @Test + void testContextIsSetDuringFilterChain() throws Exception { + final MockHttpServletRequest request = new MockHttpServletRequest(); + request.addHeader(TOKEN_HEADER, TOKEN_VALUE); + request.addHeader(ROLE_HEADER, ROLE_VALUE); + setUpSecurityContext(); + + final ConnectorRequestContext[] capturedContext = new ConnectorRequestContext[1]; + doAnswer(invocation -> { + capturedContext[0] = ConnectorRequestContextHolder.getContext(); + return null; + }).when(filterChain).doFilter(any(), any()); + + final ConnectorRequestContextFilter filter = new ConnectorRequestContextFilter(); + filter.doFilter(request, response, filterChain); + + assertNotNull(capturedContext[0]); + assertNotNull(capturedContext[0].getAuthenticatedUser()); + assertEquals(TEST_USER, capturedContext[0].getAuthenticatedUser().getIdentity()); + + assertTrue(capturedContext[0].hasRequestHeader(TOKEN_HEADER)); + assertTrue(capturedContext[0].hasRequestHeader(ROLE_HEADER)); + assertEquals(TOKEN_VALUE, capturedContext[0].getFirstRequestHeaderValue(TOKEN_HEADER)); + assertEquals(ROLE_VALUE, capturedContext[0].getFirstRequestHeaderValue(ROLE_HEADER)); + assertEquals(List.of(TOKEN_VALUE), capturedContext[0].getRequestHeaderValues(TOKEN_HEADER)); + assertEquals(List.of(ROLE_VALUE), capturedContext[0].getRequestHeaderValues(ROLE_HEADER)); + } + + @Test + void testContextIsClearedAfterFilterChain() throws Exception { + final MockHttpServletRequest request = new MockHttpServletRequest(); + request.addHeader(TOKEN_HEADER, TOKEN_VALUE); + setUpSecurityContext(); + + final ConnectorRequestContextFilter filter = new ConnectorRequestContextFilter(); + filter.doFilter(request, response, filterChain); + + assertNull(ConnectorRequestContextHolder.getContext()); + } + + @Test + void testContextIsClearedAfterFilterChainException() throws Exception { + final MockHttpServletRequest request = new MockHttpServletRequest(); + setUpSecurityContext(); + + doAnswer(invocation -> { + throw new RuntimeException("simulated error"); + }).when(filterChain).doFilter(any(), any()); + + final ConnectorRequestContextFilter filter = new ConnectorRequestContextFilter(); + try { + filter.doFilter(request, response, filterChain); + } catch (final RuntimeException ignored) { + // expected + } + + assertNull(ConnectorRequestContextHolder.getContext()); + } + + @Test + void testContextWithNoAuthenticatedUser() throws Exception { + final MockHttpServletRequest request = new MockHttpServletRequest(); + request.addHeader(TOKEN_HEADER, TOKEN_VALUE); + + final ConnectorRequestContext[] capturedContext = new ConnectorRequestContext[1]; + doAnswer(invocation -> { + capturedContext[0] = ConnectorRequestContextHolder.getContext(); + return null; + }).when(filterChain).doFilter(any(), any()); + + final ConnectorRequestContextFilter filter = new ConnectorRequestContextFilter(); + filter.doFilter(request, response, filterChain); + + assertNotNull(capturedContext[0]); + assertNull(capturedContext[0].getAuthenticatedUser()); + assertEquals(TOKEN_VALUE, capturedContext[0].getFirstRequestHeaderValue(TOKEN_HEADER)); + } + + @Test + void testHeaderMapIsCaseInsensitive() throws Exception { + final MockHttpServletRequest request = new MockHttpServletRequest(); + request.addHeader(TOKEN_HEADER, TOKEN_VALUE); + + final ConnectorRequestContext[] capturedContext = new ConnectorRequestContext[1]; + doAnswer(invocation -> { + capturedContext[0] = ConnectorRequestContextHolder.getContext(); + return null; + }).when(filterChain).doFilter(any(), any()); + + final ConnectorRequestContextFilter filter = new ConnectorRequestContextFilter(); + filter.doFilter(request, response, filterChain); + + assertNotNull(capturedContext[0]); + assertEquals(TOKEN_VALUE, capturedContext[0].getFirstRequestHeaderValue(TOKEN_HEADER.toLowerCase())); + assertEquals(TOKEN_VALUE, capturedContext[0].getFirstRequestHeaderValue(TOKEN_HEADER.toUpperCase())); + assertTrue(capturedContext[0].hasRequestHeader(TOKEN_HEADER.toLowerCase())); + assertTrue(capturedContext[0].hasRequestHeader(TOKEN_HEADER.toUpperCase())); + assertFalse(capturedContext[0].hasRequestHeader("Non-Existent-Header")); + assertEquals(List.of(), capturedContext[0].getRequestHeaderValues("Non-Existent-Header")); + assertNull(capturedContext[0].getFirstRequestHeaderValue("Non-Existent-Header")); + } + + private void setUpSecurityContext() { + final NiFiUser user = mock(NiFiUser.class, withSettings().strictness(Strictness.LENIENT)); + when(user.getIdentity()).thenReturn(TEST_USER); + final NiFiUserDetails userDetails = new NiFiUserDetails(user); + final TestingAuthenticationToken authentication = new TestingAuthenticationToken(userDetails, ""); + SecurityContextHolder.getContext().setAuthentication(authentication); + } +} diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProviderTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProviderTest.java index 3e22dc432f88..f48057a6f15f 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProviderTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProviderTest.java @@ -186,7 +186,7 @@ private NiFiProperties getProperties(final String discoveryUrl) { properties.put(NiFiProperties.SECURITY_USER_OIDC_CLIENT_SECRET, CLIENT_SECRET); properties.put(NiFiProperties.SECURITY_USER_OIDC_CLAIM_IDENTIFYING_USER, USER_NAME_ATTRIBUTE_NAME); properties.put(NiFiProperties.SECURITY_USER_OIDC_ADDITIONAL_SCOPES, OidcScopes.PROFILE); - return NiFiProperties.createBasicNiFiProperties(null, properties); + return NiFiProperties.createBasicNiFiProperties((String) null, properties); } private OIDCProviderMetadata getProviderMetadata() { diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRegistrationBuilderProviderTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRegistrationBuilderProviderTest.java index c682f316357b..5e1f5bd8b902 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRegistrationBuilderProviderTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRegistrationBuilderProviderTest.java @@ -167,7 +167,7 @@ private KeyStore getKeyStore() throws GeneralSecurityException { private NiFiProperties getProperties(final String metadataUrl) { final Properties properties = new Properties(); properties.setProperty(NiFiProperties.SECURITY_USER_SAML_IDP_METADATA_URL, metadataUrl); - return NiFiProperties.createBasicNiFiProperties(null, properties); + return NiFiProperties.createBasicNiFiProperties((String) null, properties); } private NiFiProperties getPropertiesTrustStoreStrategy(final String metadataUrl) { @@ -175,7 +175,7 @@ private NiFiProperties getPropertiesTrustStoreStrategy(final String metadataUrl) properties.setProperty(NiFiProperties.SECURITY_USER_SAML_IDP_METADATA_URL, metadataUrl); properties.setProperty(NiFiProperties.SECURITY_USER_SAML_HTTP_CLIENT_TRUSTSTORE_STRATEGY, StandardRegistrationBuilderProvider.NIFI_TRUST_STORE_STRATEGY); - return NiFiProperties.createBasicNiFiProperties(null, properties); + return NiFiProperties.createBasicNiFiProperties((String) null, properties); } final String getMetadata() throws IOException { diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRelyingPartyRegistrationRepositoryTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRelyingPartyRegistrationRepositoryTest.java index 667f8209ebd7..f3ddd48338cf 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRelyingPartyRegistrationRepositoryTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/saml2/registration/StandardRelyingPartyRegistrationRepositoryTest.java @@ -132,7 +132,7 @@ private void assertRegistrationPropertiesFound(final RelyingPartyRegistration re private NiFiProperties getProperties() { final Properties properties = getStandardProperties(); - return NiFiProperties.createBasicNiFiProperties(null, properties); + return NiFiProperties.createBasicNiFiProperties((String) null, properties); } private NiFiProperties getSingleLogoutProperties() { @@ -140,7 +140,7 @@ private NiFiProperties getSingleLogoutProperties() { properties.setProperty(NiFiProperties.SECURITY_USER_SAML_SINGLE_LOGOUT_ENABLED, Boolean.TRUE.toString()); properties.setProperty(NiFiProperties.SECURITY_USER_SAML_SIGNATURE_ALGORITHM, SignatureConstants.ALGO_ID_SIGNATURE_RSA_SHA512); - return NiFiProperties.createBasicNiFiProperties(null, properties); + return NiFiProperties.createBasicNiFiProperties((String) null, properties); } private Properties getStandardProperties() { diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/x509/X509AuthenticationProviderTest.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/x509/X509AuthenticationProviderTest.java index ac157418948e..328eab1525a0 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/x509/X509AuthenticationProviderTest.java +++ b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/java/org/apache/nifi/web/security/x509/X509AuthenticationProviderTest.java @@ -80,7 +80,7 @@ public void setup() { return AuthorizationResult.approved(); }); - x509AuthenticationProvider = new X509AuthenticationProvider(authorizer, NiFiProperties.createBasicNiFiProperties(null)); + x509AuthenticationProvider = new X509AuthenticationProvider(authorizer, NiFiProperties.createBasicNiFiProperties((String) null)); } @Test diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/component-access-policies/component-access-policies.component.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/component-access-policies/component-access-policies.component.ts index 8486e1882d00..742ba8dccc97 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/component-access-policies/component-access-policies.component.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/component-access-policies/component-access-policies.component.ts @@ -268,7 +268,8 @@ export class ComponentAccessPolicies implements OnInit, OnDestroy { } } else if ( policyComponentState.resource === 'parameter-contexts' || - policyComponentState.resource === 'parameter-providers' + policyComponentState.resource === 'parameter-providers' || + policyComponentState.resource === 'connectors' ) { switch (option.value) { case 'read-data': @@ -331,6 +332,7 @@ export class ComponentAccessPolicies implements OnInit, OnDestroy { case 'parameter-providers': case 'parameter-contexts': case 'reporting-tasks': + case 'connectors': return 'icon-drop'; } @@ -357,6 +359,8 @@ export class ComponentAccessPolicies implements OnInit, OnDestroy { return 'Reporting Task'; case 'parameter-providers': return ComponentType.ParameterProvider; + case 'connectors': + return 'Connector'; } return ComponentType.ProcessGroup; diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/global-access-policies/global-access-policies.component.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/global-access-policies/global-access-policies.component.ts index 74226feb6834..d84692cf9d47 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/global-access-policies/global-access-policies.component.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/access-policies/ui/global-access-policies/global-access-policies.component.ts @@ -201,6 +201,7 @@ export class GlobalAccessPolicies implements OnInit, OnDestroy { return ( resource === 'controller' || resource === 'parameter-contexts' || + resource === 'connectors' || resource === 'counters' || resource === 'policies' || resource === 'tenants' diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation-routing.module.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation-routing.module.ts index 544dd8106c5d..61abbbff12a3 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation-routing.module.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation-routing.module.ts @@ -26,6 +26,7 @@ import { ParameterProviderDefinition } from '../ui/parameter-provider-definition import { FlowAnalysisRuleDefinition } from '../ui/flow-analysis-rule-definition/flow-analysis-rule-definition.component'; import { Overview } from '../ui/overview/overview.component'; import { FlowRegistryClientDefinition } from '../ui/flow-registry-client-definition/flow-registry-client-definition.component'; +import { ConnectorDefinitionComponent } from '../ui/connector-definition/connector-definition.component'; const routes: Routes = [ { @@ -57,6 +58,10 @@ const routes: Routes = [ path: `${ComponentType.FlowAnalysisRule}/:group/:artifact/:version/:type`, component: FlowAnalysisRuleDefinition }, + { + path: `${ComponentType.Connector}/:group/:artifact/:version/:type`, + component: ConnectorDefinitionComponent + }, { path: 'overview', component: Overview diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.html b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.html index 22109056a224..0174272d21d5 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.html +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.html @@ -125,6 +125,19 @@ } "> + + + Connectors + + + @if (extensionTypes.length === 0) { diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.ts index ff14eb6379eb..332d9d74eb0d 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.component.ts @@ -29,6 +29,7 @@ import { NiFiState } from '../../../state'; import { Store } from '@ngrx/store'; import { loadExtensionTypesForDocumentation } from '../../../state/extension-types/extension-types.actions'; import { + selectConnectorTypes, selectControllerServiceTypes, selectFlowAnalysisRuleTypes, selectParameterProviderTypes, @@ -82,6 +83,9 @@ export class Documentation implements OnInit, AfterViewInit { flowAnalysisRuleTypes$ = this.store .select(selectFlowAnalysisRuleTypes) .pipe(map((extensionTypes) => this.sortExtensions(extensionTypes))); + connectorTypes$ = this.store + .select(selectConnectorTypes) + .pipe(map((extensionTypes) => this.sortExtensions(extensionTypes))); accordion = viewChild.required(MatAccordion); @@ -98,6 +102,7 @@ export class Documentation implements OnInit, AfterViewInit { reportingTasksExpanded = false; parameterProvidersExpanded = false; registryClientsExpanded = false; + connectorsExpanded = false; constructor() { this.store @@ -135,6 +140,9 @@ export class Documentation implements OnInit, AfterViewInit { case ComponentType.FlowRegistryClient: this.registryClientsExpanded = true; break; + case ComponentType.Connector: + this.connectorsExpanded = true; + break; } }); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.module.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.module.ts index 0766ac00a88c..5c3f0460905c 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.module.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/feature/documentation.module.ts @@ -36,6 +36,7 @@ import { ReportingTaskDefinitionEffects } from '../state/reporting-task-definiti import { ParameterProviderDefinitionEffects } from '../state/parameter-provider-definition/parameter-provider-definition.effects'; import { FlowAnalysisRuleDefinitionEffects } from '../state/flow-analysis-rule-definition/flow-analysis-rule-definition.effects'; import { FlowRegistryClientDefinitionEffects } from '../state/flow-registry-client-definition/flow-registry-client-definition.effects'; +import { ConnectorDefinitionEffects } from '../state/connector-definition/connector-definition.effects'; @NgModule({ declarations: [Documentation], @@ -51,6 +52,7 @@ import { FlowRegistryClientDefinitionEffects } from '../state/flow-registry-clie FlowRegistryClientDefinitionEffects, ParameterProviderDefinitionEffects, FlowAnalysisRuleDefinitionEffects, + ConnectorDefinitionEffects, AdditionalDetailsEffects, DocumentationEffects ), diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/service/documentation.service.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/service/documentation.service.ts index a9955bf6bda1..2b7c12f2ab24 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/service/documentation.service.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/service/documentation.service.ts @@ -26,6 +26,7 @@ import { ReportingTaskDefinition } from '../state/reporting-task-definition'; import { ParameterProviderDefinition } from '../state/parameter-provider-definition'; import { FlowAnalysisRuleDefinition } from '../state/flow-analysis-rule-definition'; import { FlowRegistryClientDefinition } from '../state/flow-registry-client-definition'; +import { ConnectorDefinition } from '../state/connector-definition'; @Injectable({ providedIn: 'root' }) export class DocumentationService { @@ -74,4 +75,20 @@ export class DocumentationService { `${DocumentationService.API}/flow/additional-details/${coordinates.group}/${coordinates.artifact}/${coordinates.version}/${coordinates.type}` ); } + + getConnectorDefinition(coordinates: DefinitionCoordinates): Observable { + return this.httpClient.get( + `${DocumentationService.API}/flow/connector-definition/${coordinates.group}/${coordinates.artifact}/${coordinates.version}/${coordinates.type}` + ); + } + + getStepDocumentation(coordinates: DefinitionCoordinates, stepName: string): Observable { + return this.httpClient.get( + `${DocumentationService.API}/flow/steps/${coordinates.group}/${coordinates.artifact}/${coordinates.version}/${coordinates.type}/${stepName}` + ); + } +} + +export interface StepDocumentationEntity { + stepDocumentation: string; } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.actions.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.actions.ts new file mode 100644 index 000000000000..aaa80ac1b1a1 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.actions.ts @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { createAction, props } from '@ngrx/store'; +import { ConnectorDefinition } from './index'; +import { DefinitionCoordinates } from '../index'; + +const CONNECTOR_DEFINITION_PREFIX = '[Connector Definition]'; + +export const loadConnectorDefinition = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Load Connector Definition`, + props<{ coordinates: DefinitionCoordinates }>() +); + +export const loadConnectorDefinitionSuccess = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Load Connector Definition Success`, + props<{ connectorDefinition: ConnectorDefinition }>() +); + +export const connectorDefinitionApiError = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Load Connector Definition Error`, + props<{ error: string }>() +); + +export const resetConnectorDefinitionState = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Reset Connector Definition State` +); + +export const loadStepDocumentation = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Load Step Documentation`, + props<{ coordinates: DefinitionCoordinates; stepName: string }>() +); + +export const loadStepDocumentationSuccess = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Load Step Documentation Success`, + props<{ stepName: string; documentation: string }>() +); + +export const stepDocumentationApiError = createAction( + `${CONNECTOR_DEFINITION_PREFIX} Load Step Documentation Error`, + props<{ stepName: string; error: string }>() +); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.effects.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.effects.ts new file mode 100644 index 000000000000..e4ebac8ffcd2 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.effects.ts @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Injectable, inject } from '@angular/core'; +import { Actions, createEffect, ofType } from '@ngrx/effects'; +import * as ConnectorDefinitionActions from './connector-definition.actions'; +import { catchError, from, map, mergeMap, of, switchMap } from 'rxjs'; +import { DocumentationService } from '../../service/documentation.service'; +import { ConnectorDefinition } from './index'; +import { ErrorHelper } from '../../../../service/error-helper.service'; +import { HttpErrorResponse } from '@angular/common/http'; + +@Injectable() +export class ConnectorDefinitionEffects { + private actions$ = inject(Actions); + private documentationService = inject(DocumentationService); + private errorHelper = inject(ErrorHelper); + + loadConnectorDefinition$ = createEffect(() => + this.actions$.pipe( + ofType(ConnectorDefinitionActions.loadConnectorDefinition), + map((action) => action.coordinates), + switchMap((coordinates) => + from(this.documentationService.getConnectorDefinition(coordinates)).pipe( + map((connectorDefinition: ConnectorDefinition) => + ConnectorDefinitionActions.loadConnectorDefinitionSuccess({ + connectorDefinition + }) + ), + catchError((errorResponse: HttpErrorResponse) => + of( + ConnectorDefinitionActions.connectorDefinitionApiError({ + error: this.errorHelper.getErrorString(errorResponse) + }) + ) + ) + ) + ) + ) + ); + + loadStepDocumentation$ = createEffect(() => + this.actions$.pipe( + ofType(ConnectorDefinitionActions.loadStepDocumentation), + mergeMap((action) => + from(this.documentationService.getStepDocumentation(action.coordinates, action.stepName)).pipe( + map((response) => + ConnectorDefinitionActions.loadStepDocumentationSuccess({ + stepName: action.stepName, + documentation: response.stepDocumentation + }) + ), + catchError((errorResponse: HttpErrorResponse) => + of( + ConnectorDefinitionActions.stepDocumentationApiError({ + stepName: action.stepName, + error: this.errorHelper.getErrorString(errorResponse) + }) + ) + ) + ) + ) + ) + ); +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.reducer.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.reducer.ts new file mode 100644 index 000000000000..e2953573dd51 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.reducer.ts @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { createReducer, on } from '@ngrx/store'; +import { ConnectorDefinitionState } from './index'; +import { + connectorDefinitionApiError, + loadConnectorDefinition, + loadConnectorDefinitionSuccess, + loadStepDocumentation, + loadStepDocumentationSuccess, + resetConnectorDefinitionState, + stepDocumentationApiError +} from './connector-definition.actions'; + +export const initialConnectorDefinitionState: ConnectorDefinitionState = { + connectorDefinition: null, + error: null, + status: 'pending', + stepDocumentation: {} +}; + +export const connectorDefinitionReducer = createReducer( + initialConnectorDefinitionState, + on(loadConnectorDefinition, (state) => ({ + ...state, + connectorDefinition: null, + error: null, + status: 'loading' as const + })), + on(loadConnectorDefinitionSuccess, (state, { connectorDefinition }) => ({ + ...state, + connectorDefinition, + error: null, + status: 'success' as const + })), + on(connectorDefinitionApiError, (state, { error }) => ({ + ...state, + error, + status: 'error' as const + })), + on(resetConnectorDefinitionState, () => ({ + ...initialConnectorDefinitionState + })), + on(loadStepDocumentation, (state, { stepName }) => ({ + ...state, + stepDocumentation: { + ...state.stepDocumentation, + [stepName]: { + documentation: null, + error: null, + status: 'loading' as const + } + } + })), + on(loadStepDocumentationSuccess, (state, { stepName, documentation }) => ({ + ...state, + stepDocumentation: { + ...state.stepDocumentation, + [stepName]: { + documentation, + error: null, + status: 'success' as const + } + } + })), + on(stepDocumentationApiError, (state, { stepName, error }) => ({ + ...state, + stepDocumentation: { + ...state.stepDocumentation, + [stepName]: { + documentation: null, + error, + status: 'error' as const + } + } + })) +); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.selectors.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.selectors.ts new file mode 100644 index 000000000000..ad0adc5060cb --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/connector-definition.selectors.ts @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { createSelector } from '@ngrx/store'; +import { DocumentationState, selectDocumentationState } from '../index'; +import { connectorDefinitionFeatureKey, StepDocumentationState } from './index'; + +export const selectConnectorDefinitionState = createSelector( + selectDocumentationState, + (state: DocumentationState) => state[connectorDefinitionFeatureKey] +); + +export const selectStepDocumentation = (stepName: string) => + createSelector( + selectConnectorDefinitionState, + (state): StepDocumentationState | undefined => state.stepDocumentation[stepName] + ); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/index.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/index.ts new file mode 100644 index 000000000000..fda3d93c8b3f --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/connector-definition/index.ts @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { ExtensionComponent, PropertyAllowableValue } from '../index'; + +export const connectorDefinitionFeatureKey = 'connectorDefinition'; + +export interface ConnectorPropertyDependency { + propertyName: string; + dependentValues?: string[]; +} + +export interface ConnectorPropertyDescriptor { + name: string; + description: string; + defaultValue?: string; + required: boolean; + propertyType?: string; + allowableValuesFetchable: boolean; + allowableValues?: PropertyAllowableValue[]; + dependencies?: ConnectorPropertyDependency[]; +} + +export interface ConnectorPropertyGroup { + name: string; + description: string; + properties?: ConnectorPropertyDescriptor[]; +} + +export interface ConfigurationStepDependency { + stepName: string; + propertyName: string; + dependentValues?: string[]; +} + +export interface ConfigurationStep { + name: string; + description: string; + documented: boolean; + stepDependencies?: ConfigurationStepDependency[]; + propertyGroups?: ConnectorPropertyGroup[]; +} + +export interface ConnectorDefinition extends ExtensionComponent { + configurationSteps?: ConfigurationStep[]; +} + +export interface StepDocumentationState { + documentation: string | null; + error: string | null; + status: 'pending' | 'loading' | 'success' | 'error'; +} + +export interface ConnectorDefinitionState { + connectorDefinition: ConnectorDefinition | null; + error: string | null; + status: 'pending' | 'loading' | 'success' | 'error'; + stepDocumentation: { [stepName: string]: StepDocumentationState }; +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/index.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/index.ts index 0e81eab6441c..65d380edb785 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/index.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/state/index.ts @@ -41,6 +41,8 @@ import { import { parameterProviderDefinitionReducer } from './parameter-provider-definition/parameter-provider-definition.reducer'; import { flowAnalysisRuleDefinitionFeatureKey, FlowAnalysisRuleDefinitionState } from './flow-analysis-rule-definition'; import { flowAnalysisRuleDefinitionReducer } from './flow-analysis-rule-definition/flow-analysis-rule-definition.reducer'; +import { connectorDefinitionFeatureKey, ConnectorDefinitionState } from './connector-definition'; +import { connectorDefinitionReducer } from './connector-definition/connector-definition.reducer'; import { ComponentType } from '@nifi/shared'; import { DocumentedType } from '../../../state/shared'; @@ -196,6 +198,7 @@ export interface DocumentationState { [flowRegistryClientDefinitionFeatureKey]: FlowRegistryClientDefinitionState; [parameterProviderDefinitionFeatureKey]: ParameterProviderDefinitionState; [flowAnalysisRuleDefinitionFeatureKey]: FlowAnalysisRuleDefinitionState; + [connectorDefinitionFeatureKey]: ConnectorDefinitionState; [additionalDetailsFeatureKey]: AdditionalDetailsState; [externalDocumentationFeatureKey]: ExternalDocumentationState; } @@ -208,6 +211,7 @@ export function reducers(state: DocumentationState | undefined, action: Action) [flowRegistryClientDefinitionFeatureKey]: flowRegistryClientDefinitionReducer, [parameterProviderDefinitionFeatureKey]: parameterProviderDefinitionReducer, [flowAnalysisRuleDefinitionFeatureKey]: flowAnalysisRuleDefinitionReducer, + [connectorDefinitionFeatureKey]: connectorDefinitionReducer, [additionalDetailsFeatureKey]: additionalDetailsReducer, [externalDocumentationFeatureKey]: externalDocumentationReducer })(state, action); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.html b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.html new file mode 100644 index 000000000000..395c7fff9df8 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.html @@ -0,0 +1,95 @@ + + +@if (propertyDescriptor) { +
+
+
Name
+
{{ propertyDescriptor.name }}
+
+
+
Description
+
{{ propertyDescriptor.description }}
+
+
+
Type
+
{{ formatPropertyType() }}
+
+
+
Required
+
{{ propertyDescriptor.required ? 'Yes' : 'No' }}
+
+ @if (formatDefaultValue(); as defaultValue) { +
+
Default Value
+
{{ defaultValue }}
+
+ } + @if (propertyDescriptor.allowableValues) { +
+
Allowable Values
+
+
    + @for (allowableValue of propertyDescriptor.allowableValues; track allowableValue.value) { +
  • + {{ allowableValue.displayName }} + @if (allowableValue.description) { + + } +
    • + } +
+
+
+ } + @if (propertyDescriptor.dependencies) { +
+
+ Dependencies + +
+
+
    + @for ( + propertyDependency of sortDependencies(propertyDescriptor.dependencies); + track propertyDependency.propertyName + ) { +
  • + @if (propertyDependency.dependentValues) { + {{ propertyDependency.propertyName }} is set to any of [{{ + formatDependentValue(propertyDependency) + }}] + } @else { + {{ propertyDependency.propertyName }} has any value specified + } +
    • + } +
+
+
+ } +
+} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.scss b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.scss new file mode 100644 index 000000000000..eb0ceabcbddf --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.scss @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +:host { + display: block; +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.spec.ts new file mode 100644 index 000000000000..45143688eb52 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.spec.ts @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { TestBed } from '@angular/core/testing'; + +import { ConnectorPropertyDefinitionComponent } from './connector-property-definition.component'; +import { ConnectorPropertyDescriptor } from '../../../state/connector-definition'; + +describe('ConnectorPropertyDefinitionComponent', () => { + function createMockProperty(overrides?: Partial): ConnectorPropertyDescriptor { + return { + name: 'Test Property', + description: 'A test property description', + required: false, + allowableValuesFetchable: false, + ...overrides + }; + } + + async function setup(propertyOverrides?: Partial) { + await TestBed.configureTestingModule({ + imports: [ConnectorPropertyDefinitionComponent] + }).compileComponents(); + + const fixture = TestBed.createComponent(ConnectorPropertyDefinitionComponent); + const component = fixture.componentInstance; + component.propertyDescriptor = createMockProperty(propertyOverrides); + fixture.detectChanges(); + + return { component, fixture }; + } + + describe('Component initialization', () => { + it('should create', async () => { + const { component } = await setup(); + expect(component).toBeTruthy(); + }); + + it('should have property descriptor set', async () => { + const { component } = await setup({ name: 'Custom Property' }); + expect(component.propertyDescriptor.name).toBe('Custom Property'); + }); + }); + + describe('Default value formatting', () => { + it('should return undefined when no default value is set', async () => { + const { component } = await setup({ defaultValue: undefined }); + expect(component.formatDefaultValue()).toBeUndefined(); + }); + + it('should return the default value as-is when no allowable values exist', async () => { + const { component } = await setup({ defaultValue: 'my-default' }); + expect(component.formatDefaultValue()).toBe('my-default'); + }); + + it('should return the display name when default value matches an allowable value', async () => { + const { component } = await setup({ + defaultValue: 'val1', + allowableValues: [ + { value: 'val1', displayName: 'Value One', description: 'First value' }, + { value: 'val2', displayName: 'Value Two', description: 'Second value' } + ] + }); + expect(component.formatDefaultValue()).toBe('Value One'); + }); + + it('should return the raw default value when it does not match any allowable value', async () => { + const { component } = await setup({ + defaultValue: 'unknown', + allowableValues: [{ value: 'val1', displayName: 'Value One', description: 'First value' }] + }); + expect(component.formatDefaultValue()).toBe('unknown'); + }); + }); + + describe('Property type formatting', () => { + it('should return STRING when property type is undefined', async () => { + const { component } = await setup({ propertyType: undefined }); + expect(component.formatPropertyType()).toBe('STRING'); + }); + + it('should return the property type when defined', async () => { + const { component } = await setup({ propertyType: 'INTEGER' }); + expect(component.formatPropertyType()).toBe('INTEGER'); + }); + }); + + describe('Dependency sorting', () => { + it('should sort dependencies alphabetically by property name', async () => { + const { component } = await setup(); + const dependencies = [ + { propertyName: 'zeta', dependentValues: ['a'] }, + { propertyName: 'alpha', dependentValues: ['b'] }, + { propertyName: 'gamma', dependentValues: ['c'] } + ]; + + const sorted = component.sortDependencies(dependencies); + + expect(sorted[0].propertyName).toBe('alpha'); + expect(sorted[1].propertyName).toBe('gamma'); + expect(sorted[2].propertyName).toBe('zeta'); + }); + + it('should not mutate the original array', async () => { + const { component } = await setup(); + const dependencies = [ + { propertyName: 'zeta', dependentValues: ['a'] }, + { propertyName: 'alpha', dependentValues: ['b'] } + ]; + + component.sortDependencies(dependencies); + + expect(dependencies[0].propertyName).toBe('zeta'); + }); + }); + + describe('Dependent value formatting', () => { + it('should return empty string when dependentValues is undefined', async () => { + const { component } = await setup(); + const dependency = { propertyName: 'prop1' }; + + expect(component.formatDependentValue(dependency)).toBe(''); + }); + + it('should join dependent values with comma', async () => { + const { component } = await setup(); + const dependency = { propertyName: 'prop1', dependentValues: ['val1', 'val2', 'val3'] }; + + expect(component.formatDependentValue(dependency)).toBe('val1, val2, val3'); + }); + + it('should use display names from lookupProperty when available', async () => { + const { component } = await setup(); + const properties: ConnectorPropertyDescriptor[] = [ + { + name: 'dep-prop', + description: 'Dependency property', + required: false, + allowableValuesFetchable: false, + allowableValues: [ + { value: 'v1', displayName: 'Display Value 1', description: '' }, + { value: 'v2', displayName: 'Display Value 2', description: '' } + ] + } + ]; + component.lookupProperty = (name: string) => properties.find((p) => p.name === name); + + const dependency = { propertyName: 'dep-prop', dependentValues: ['v1', 'v2'] }; + + expect(component.formatDependentValue(dependency)).toBe('Display Value 1, Display Value 2'); + }); + + it('should fall back to raw values when lookupProperty does not find the property', async () => { + const { component } = await setup(); + component.lookupProperty = () => undefined; + + const dependency = { propertyName: 'unknown-prop', dependentValues: ['raw1', 'raw2'] }; + + expect(component.formatDependentValue(dependency)).toBe('raw1, raw2'); + }); + }); +}); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.ts new file mode 100644 index 000000000000..fae2f0094511 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/common/connector-property-definition/connector-property-definition.component.ts @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Component, Input, inject } from '@angular/core'; +import { NiFiCommon, NifiTooltipDirective, TextTip } from '@nifi/shared'; +import { ConnectorPropertyDescriptor, ConnectorPropertyDependency } from '../../../state/connector-definition'; +import { PropertyAllowableValue } from '../../../state'; + +@Component({ + selector: 'connector-property-definition', + imports: [NifiTooltipDirective], + templateUrl: './connector-property-definition.component.html', + styleUrl: './connector-property-definition.component.scss' +}) +export class ConnectorPropertyDefinitionComponent { + private nifiCommon = inject(NiFiCommon); + + @Input() propertyDescriptor!: ConnectorPropertyDescriptor; + @Input() lookupProperty: ((name: string) => ConnectorPropertyDescriptor | undefined) | undefined; + + protected readonly TextTip = TextTip; + + formatDefaultValue(): string | undefined { + if (!this.propertyDescriptor.defaultValue) { + return undefined; + } + + if (this.propertyDescriptor.allowableValues) { + const defaultAllowableValue: PropertyAllowableValue | undefined = + this.propertyDescriptor.allowableValues.find( + (allowableValue) => allowableValue.value === this.propertyDescriptor.defaultValue + ); + + if (defaultAllowableValue) { + return defaultAllowableValue.displayName; + } + } + + return this.propertyDescriptor.defaultValue; + } + + formatPropertyType(): string { + return this.propertyDescriptor.propertyType || 'STRING'; + } + + sortDependencies(dependencies: ConnectorPropertyDependency[]): ConnectorPropertyDependency[] { + return dependencies.slice().sort((a, b) => { + return this.nifiCommon.compareString(a.propertyName, b.propertyName); + }); + } + + formatDependentValue(dependency: ConnectorPropertyDependency): string { + if (dependency.dependentValues) { + if (this.lookupProperty) { + const propertyDependency = this.lookupProperty(dependency.propertyName); + + if (propertyDependency?.allowableValues) { + return dependency.dependentValues + .map((dependentValue) => { + const dependentAllowableValue = propertyDependency.allowableValues?.find( + (allowableValue) => dependentValue === allowableValue.value + ); + return dependentAllowableValue ? dependentAllowableValue.displayName : dependentValue; + }) + .join(', '); + } + } + + return dependency.dependentValues.join(', '); + } + + return ''; + } +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.html b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.html new file mode 100644 index 000000000000..849ce1db7310 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.html @@ -0,0 +1,209 @@ + + +@if (connectorDefinitionState) { + @if (isInitialLoading(connectorDefinitionState)) { + + } @else { + @if (connectorDefinitionState.connectorDefinition; as connectorDefinition) { +
+

+ {{ formatExtensionName(connectorDefinition.type) }} {{ connectorDefinition.version }} +

+
+ @if (connectorDefinition.deprecated) { +
+ @if (connectorDefinition.deprecationReason) { +
{{ connectorDefinition.deprecationReason }}
+ } @else { +
+ Please be aware this connector is deprecated and may be removed in the near future. +
+ } + @if (connectorDefinition.deprecationAlternatives) { +
+
See Alternatives
+ +
+ } +
+ } +
+
Bundle
+
+ {{ connectorDefinition.group }} | {{ connectorDefinition.artifact }} +
+
+ @if (connectorDefinition.typeDescription) { +
+
Description
+
{{ connectorDefinition.typeDescription }}
+
+ } + @if (connectorDefinition.tags) { +
+
Tags
+
+ {{ connectorDefinition.tags.join(', ') }} +
+
+ } + @if (hasConfigurationSteps(connectorDefinition.configurationSteps)) { +
+
+

Configuration Steps

+
+ + +
+
+ + @for ( + step of connectorDefinition.configurationSteps; + track step.name; + let first = $first + ) { + + + {{ step.name }} + +
+ @if (step.description) { +
{{ step.description }}
+ } + @if (step.documented) { + + + Step Documentation + +
+ @if (isStepDocumentationLoading(step.name)) { + + } @else if (getStepDocumentationError(step.name); as error) { +
{{ error }}
+ } @else if (getStepDocumentation(step.name); as documentation) { + + } +
+ + } + @if (hasPropertyGroups(step.propertyGroups)) { + @for (group of step.propertyGroups; track group.name) { +
+
+

{{ group.name }}

+
+ @if (group.description) { +
+ {{ group.description }} +
+ } + @if (hasProperties(group.properties)) { +
+
+ In the list below required properties are shown with + an asterisk (*). Other properties are considered + optional. +
+
+ + +
+
+ + @for ( + property of group.properties; + track property.name + ) { + + + + {{ formatPropertyTitle(property) }} + + + {{ property.description }} + + + + + } + + } +
+ } + } +
+ + } + +
+ } + @if (connectorDefinition.seeAlso) { +
+

See Also

+ +
+ } +
+
+ } @else if (connectorDefinitionState.error) { +
+ {{ connectorDefinitionState.error }} +
+ } + } +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.scss b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.scss new file mode 100644 index 000000000000..eb0ceabcbddf --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.scss @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +:host { + display: block; +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.spec.ts new file mode 100644 index 000000000000..41ac8cac5f86 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.spec.ts @@ -0,0 +1,415 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { TestBed } from '@angular/core/testing'; +import { MockStore, provideMockStore } from '@ngrx/store/testing'; + +import { ConnectorDefinitionComponent } from './connector-definition.component'; +import { + ConfigurationStep, + connectorDefinitionFeatureKey, + ConnectorDefinitionState, + ConnectorPropertyDescriptor, + ConnectorPropertyGroup +} from '../../state/connector-definition'; +import { initialConnectorDefinitionState } from '../../state/connector-definition/connector-definition.reducer'; +import { documentationFeatureKey } from '../../state'; +import { initialState as initialErrorState } from '../../../../state/error/error.reducer'; +import { errorFeatureKey } from '../../../../state/error'; +import { initialState as initialCurrentUserState } from '../../../../state/current-user/current-user.reducer'; +import { currentUserFeatureKey } from '../../../../state/current-user'; +import { ConnectorDefinition } from '../../state/connector-definition'; +import { loadStepDocumentation } from '../../state/connector-definition/connector-definition.actions'; + +describe('ConnectorDefinitionComponent', () => { + function createMockProperty(name: string, required: boolean, description: string): ConnectorPropertyDescriptor { + return { + name, + description, + required, + allowableValuesFetchable: false + }; + } + + function createMockPropertyGroup(name: string, properties: ConnectorPropertyDescriptor[]): ConnectorPropertyGroup { + return { + name, + description: `Description for ${name}`, + properties + }; + } + + function createMockConfigurationStep(name: string, propertyGroups: ConnectorPropertyGroup[]): ConfigurationStep { + return { + name, + description: `Description for ${name}`, + documented: false, + propertyGroups + }; + } + + async function setup(stateOverrides?: Partial) { + const initialState = { + [errorFeatureKey]: initialErrorState, + [currentUserFeatureKey]: initialCurrentUserState, + [documentationFeatureKey]: { + [connectorDefinitionFeatureKey]: { + ...initialConnectorDefinitionState, + ...stateOverrides + } + } + }; + + await TestBed.configureTestingModule({ + imports: [ConnectorDefinitionComponent], + providers: [provideMockStore({ initialState })] + }).compileComponents(); + + const fixture = TestBed.createComponent(ConnectorDefinitionComponent); + const component = fixture.componentInstance; + const store = TestBed.inject(MockStore); + + fixture.detectChanges(); + + return { component, fixture, store }; + } + + describe('Component initialization', () => { + it('should create', async () => { + const { component } = await setup(); + expect(component).toBeTruthy(); + }); + + it('should have null connectorDefinitionState initially', async () => { + const { component } = await setup(); + expect(component.connectorDefinitionState).toBeDefined(); + }); + }); + + describe('Loading state logic', () => { + it('should return true for isInitialLoading when connectorDefinition and error are null', async () => { + const { component } = await setup(); + const state: ConnectorDefinitionState = { + connectorDefinition: null, + error: null, + status: 'pending', + stepDocumentation: {} + }; + expect(component.isInitialLoading(state)).toBe(true); + }); + + it('should return false for isInitialLoading when connectorDefinition is present', async () => { + const { component } = await setup(); + const state: ConnectorDefinitionState = { + connectorDefinition: { + group: 'org.apache.nifi', + artifact: 'nifi-test-nar', + version: '1.0.0', + type: 'org.apache.nifi.TestConnector', + typeDescription: 'Test Connector', + buildInfo: { + revision: 'abc123', + version: '1.0.0' + }, + additionalDetails: false + }, + error: null, + status: 'success', + stepDocumentation: {} + }; + expect(component.isInitialLoading(state)).toBe(false); + }); + + it('should return false for isInitialLoading when error is present', async () => { + const { component } = await setup(); + const state: ConnectorDefinitionState = { + connectorDefinition: null, + error: 'Failed to load', + status: 'error', + stepDocumentation: {} + }; + expect(component.isInitialLoading(state)).toBe(false); + }); + }); + + describe('Configuration step helpers', () => { + it('should return true for hasConfigurationSteps when steps array has items', async () => { + const { component } = await setup(); + const steps = [createMockConfigurationStep('Step 1', [])]; + expect(component.hasConfigurationSteps(steps)).toBe(true); + }); + + it('should return false for hasConfigurationSteps when steps array is empty', async () => { + const { component } = await setup(); + expect(component.hasConfigurationSteps([])).toBe(false); + }); + + it('should return false for hasConfigurationSteps when steps is undefined', async () => { + const { component } = await setup(); + expect(component.hasConfigurationSteps(undefined)).toBe(false); + }); + }); + + describe('Property group helpers', () => { + it('should return true for hasPropertyGroups when groups array has items', async () => { + const { component } = await setup(); + const groups = [createMockPropertyGroup('Group 1', [])]; + expect(component.hasPropertyGroups(groups)).toBe(true); + }); + + it('should return false for hasPropertyGroups when groups array is empty', async () => { + const { component } = await setup(); + expect(component.hasPropertyGroups([])).toBe(false); + }); + + it('should return false for hasPropertyGroups when groups is undefined', async () => { + const { component } = await setup(); + expect(component.hasPropertyGroups(undefined)).toBe(false); + }); + }); + + describe('Property helpers', () => { + it('should return true for hasProperties when properties array has items', async () => { + const { component } = await setup(); + const properties = [createMockProperty('prop1', true, 'Description')]; + expect(component.hasProperties(properties)).toBe(true); + }); + + it('should return false for hasProperties when properties array is empty', async () => { + const { component } = await setup(); + expect(component.hasProperties([])).toBe(false); + }); + + it('should return false for hasProperties when properties is undefined', async () => { + const { component } = await setup(); + expect(component.hasProperties(undefined)).toBe(false); + }); + }); + + describe('Property title formatting', () => { + it('should append asterisk for required properties', async () => { + const { component } = await setup(); + const property = createMockProperty('Required Property', true, 'Description'); + expect(component.formatPropertyTitle(property)).toBe('Required Property*'); + }); + + it('should not append asterisk for optional properties', async () => { + const { component } = await setup(); + const property = createMockProperty('Optional Property', false, 'Description'); + expect(component.formatPropertyTitle(property)).toBe('Optional Property'); + }); + }); + + describe('Property lookup', () => { + it('should return a function that finds properties by name', async () => { + const { component } = await setup(); + const properties = [ + createMockProperty('prop1', true, 'First property'), + createMockProperty('prop2', false, 'Second property') + ]; + + const lookupFn = component.lookupProperty(properties); + + expect(lookupFn('prop1')).toEqual(properties[0]); + expect(lookupFn('prop2')).toEqual(properties[1]); + expect(lookupFn('nonexistent')).toBeUndefined(); + }); + }); + + describe('Step documentation', () => { + it('should return false for isStepDocumentationLoading when step is not in state', async () => { + const { component } = await setup(); + expect(component.isStepDocumentationLoading('Some Step')).toBe(false); + }); + + it('should return true for isStepDocumentationLoading when step status is loading', async () => { + const { component } = await setup({ + stepDocumentation: { + 'Some Step': { documentation: null, error: null, status: 'loading' } + } + }); + expect(component.isStepDocumentationLoading('Some Step')).toBe(true); + }); + + it('should return undefined for getStepDocumentation when step is not in state', async () => { + const { component } = await setup(); + expect(component.getStepDocumentation('Some Step')).toBeUndefined(); + }); + + it('should return documentation when step has loaded documentation', async () => { + const { component } = await setup({ + stepDocumentation: { + 'Some Step': { documentation: '# Step Documentation', error: null, status: 'success' } + } + }); + expect(component.getStepDocumentation('Some Step')).toBe('# Step Documentation'); + }); + + it('should return undefined for getStepDocumentationError when step is not in state', async () => { + const { component } = await setup(); + expect(component.getStepDocumentationError('Some Step')).toBeUndefined(); + }); + + it('should return error when step has an error', async () => { + const { component } = await setup({ + stepDocumentation: { + 'Some Step': { documentation: null, error: 'Failed to load', status: 'error' } + } + }); + expect(component.getStepDocumentationError('Some Step')).toBe('Failed to load'); + }); + }); + + describe('onStepExpanded', () => { + function createMockConnectorDefinition(): ConnectorDefinition { + return { + group: 'org.apache.nifi', + artifact: 'nifi-test-nar', + version: '1.0.0', + type: 'org.apache.nifi.TestConnector', + typeDescription: 'Test Connector', + buildInfo: { revision: 'abc123', version: '1.0.0' }, + additionalDetails: false + }; + } + + it('should call loadStepDocumentation when step is documented', async () => { + const { component, store } = await setup(); + const dispatchSpy = jest.spyOn(store, 'dispatch'); + const connectorDefinition = createMockConnectorDefinition(); + const step = createMockConfigurationStep('Documented Step', []); + step.documented = true; + + component.onStepExpanded(connectorDefinition, step); + + expect(dispatchSpy).toHaveBeenCalledWith( + loadStepDocumentation({ + coordinates: { + group: connectorDefinition.group, + artifact: connectorDefinition.artifact, + version: connectorDefinition.version, + type: connectorDefinition.type + }, + stepName: 'Documented Step' + }) + ); + }); + + it('should not call loadStepDocumentation when step is not documented', async () => { + const { component, store } = await setup(); + const dispatchSpy = jest.spyOn(store, 'dispatch'); + dispatchSpy.mockClear(); + const connectorDefinition = createMockConnectorDefinition(); + const step = createMockConfigurationStep('Undocumented Step', []); + step.documented = false; + + component.onStepExpanded(connectorDefinition, step); + + expect(dispatchSpy).not.toHaveBeenCalled(); + }); + }); + + describe('loadStepDocumentation deduplication', () => { + function createMockConnectorDefinition(): ConnectorDefinition { + return { + group: 'org.apache.nifi', + artifact: 'nifi-test-nar', + version: '1.0.0', + type: 'org.apache.nifi.TestConnector', + typeDescription: 'Test Connector', + buildInfo: { revision: 'abc123', version: '1.0.0' }, + additionalDetails: false + }; + } + + it('should not dispatch when step is already loading', async () => { + const { component, store } = await setup({ + stepDocumentation: { + 'Loading Step': { documentation: null, error: null, status: 'loading' } + } + }); + const dispatchSpy = jest.spyOn(store, 'dispatch'); + dispatchSpy.mockClear(); + const connectorDefinition = createMockConnectorDefinition(); + + component.loadStepDocumentation(connectorDefinition, 'Loading Step'); + + expect(dispatchSpy).not.toHaveBeenCalled(); + }); + + it('should not dispatch when step already has documentation', async () => { + const { component, store } = await setup({ + stepDocumentation: { + 'Loaded Step': { documentation: '# Docs', error: null, status: 'success' } + } + }); + const dispatchSpy = jest.spyOn(store, 'dispatch'); + dispatchSpy.mockClear(); + const connectorDefinition = createMockConnectorDefinition(); + + component.loadStepDocumentation(connectorDefinition, 'Loaded Step'); + + expect(dispatchSpy).not.toHaveBeenCalled(); + }); + + it('should dispatch when step is not in state', async () => { + const { component, store } = await setup(); + const dispatchSpy = jest.spyOn(store, 'dispatch'); + dispatchSpy.mockClear(); + const connectorDefinition = createMockConnectorDefinition(); + + component.loadStepDocumentation(connectorDefinition, 'New Step'); + + expect(dispatchSpy).toHaveBeenCalledWith( + loadStepDocumentation({ + coordinates: { + group: connectorDefinition.group, + artifact: connectorDefinition.artifact, + version: connectorDefinition.version, + type: connectorDefinition.type + }, + stepName: 'New Step' + }) + ); + }); + + it('should dispatch when step had an error', async () => { + const { component, store } = await setup({ + stepDocumentation: { + 'Error Step': { documentation: null, error: 'Failed', status: 'error' } + } + }); + const dispatchSpy = jest.spyOn(store, 'dispatch'); + dispatchSpy.mockClear(); + const connectorDefinition = createMockConnectorDefinition(); + + component.loadStepDocumentation(connectorDefinition, 'Error Step'); + + expect(dispatchSpy).toHaveBeenCalledWith( + loadStepDocumentation({ + coordinates: { + group: connectorDefinition.group, + artifact: connectorDefinition.artifact, + version: connectorDefinition.version, + type: connectorDefinition.type + }, + stepName: 'Error Step' + }) + ); + }); + }); +}); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.ts new file mode 100644 index 000000000000..7e57bb0dea89 --- /dev/null +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/documentation/ui/connector-definition/connector-definition.component.ts @@ -0,0 +1,204 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Component, OnDestroy, ViewChild, inject } from '@angular/core'; +import { Store } from '@ngrx/store'; +import { NiFiState } from '../../../../state'; +import { NgxSkeletonLoaderModule } from 'ngx-skeleton-loader'; +import { ComponentType, isDefinedAndNotNull, NiFiCommon } from '@nifi/shared'; +import { takeUntilDestroyed } from '@angular/core/rxjs-interop'; +import { selectDefinitionCoordinatesFromRouteForComponentType } from '../../state/documentation/documentation.selectors'; +import { distinctUntilChanged } from 'rxjs'; +import { + ConnectorDefinitionState, + ConfigurationStep, + ConnectorPropertyGroup, + ConnectorPropertyDescriptor, + ConnectorDefinition, + StepDocumentationState +} from '../../state/connector-definition'; +import { + loadConnectorDefinition, + loadStepDocumentation, + resetConnectorDefinitionState +} from '../../state/connector-definition/connector-definition.actions'; +import { selectConnectorDefinitionState } from '../../state/connector-definition/connector-definition.selectors'; +import { MatAccordion, MatExpansionModule } from '@angular/material/expansion'; +import { MatButtonModule } from '@angular/material/button'; +import { SeeAlsoComponent } from '../common/see-also/see-also.component'; +import { MarkdownComponent } from 'ngx-markdown'; +import { ConnectorPropertyDefinitionComponent } from '../common/connector-property-definition/connector-property-definition.component'; + +@Component({ + selector: 'connector-definition', + imports: [ + NgxSkeletonLoaderModule, + MatExpansionModule, + MatButtonModule, + SeeAlsoComponent, + MarkdownComponent, + ConnectorPropertyDefinitionComponent + ], + templateUrl: './connector-definition.component.html', + styleUrl: './connector-definition.component.scss' +}) +export class ConnectorDefinitionComponent implements OnDestroy { + private store = inject>(Store); + private nifiCommon = inject(NiFiCommon); + + @ViewChild('stepsAccordion') stepsAccordion!: MatAccordion; + + connectorDefinitionState: ConnectorDefinitionState | null = null; + + constructor() { + this.store + .select(selectDefinitionCoordinatesFromRouteForComponentType(ComponentType.Connector)) + .pipe( + isDefinedAndNotNull(), + distinctUntilChanged( + (a, b) => + a.group === b.group && a.artifact === b.artifact && a.version === b.version && a.type === b.type + ), + takeUntilDestroyed() + ) + .subscribe((coordinates) => { + this.store.dispatch( + loadConnectorDefinition({ + coordinates + }) + ); + }); + + this.store + .select(selectConnectorDefinitionState) + .pipe(takeUntilDestroyed()) + .subscribe((connectorDefinitionState) => { + const previousState = this.connectorDefinitionState; + this.connectorDefinitionState = connectorDefinitionState; + + if (connectorDefinitionState.status === 'loading') { + window.scrollTo({ top: 0, left: 0 }); + } + + if ( + previousState?.status !== 'success' && + connectorDefinitionState.status === 'success' && + connectorDefinitionState.connectorDefinition + ) { + const firstStep = connectorDefinitionState.connectorDefinition.configurationSteps?.[0]; + if (firstStep?.documented) { + this.loadStepDocumentation(connectorDefinitionState.connectorDefinition, firstStep.name); + } + } + }); + } + + isInitialLoading(state: ConnectorDefinitionState): boolean { + return state.connectorDefinition === null && state.error === null; + } + + formatExtensionName(type: string): string { + return this.nifiCommon.getComponentTypeLabel(type); + } + + hasConfigurationSteps(steps: ConfigurationStep[] | undefined): boolean { + return steps !== undefined && steps.length > 0; + } + + hasPropertyGroups(groups: ConnectorPropertyGroup[] | undefined): boolean { + return groups !== undefined && groups.length > 0; + } + + hasProperties(properties: ConnectorPropertyDescriptor[] | undefined): boolean { + return properties !== undefined && properties.length > 0; + } + + formatPropertyTitle(descriptor: ConnectorPropertyDescriptor): string { + if (descriptor.required) { + return `${descriptor.name}*`; + } + return descriptor.name; + } + + lookupProperty( + properties: ConnectorPropertyDescriptor[] + ): (name: string) => ConnectorPropertyDescriptor | undefined { + return (name: string) => properties.find((prop) => prop.name === name); + } + + expandAllSteps(): void { + this.stepsAccordion.openAll(); + } + + collapseAllSteps(): void { + this.stepsAccordion.closeAll(); + } + + expandAllProperties(accordion: MatAccordion): void { + accordion.openAll(); + } + + collapseAllProperties(accordion: MatAccordion): void { + accordion.closeAll(); + } + + onStepExpanded(connectorDefinition: ConnectorDefinition, step: ConfigurationStep): void { + if (step.documented) { + this.loadStepDocumentation(connectorDefinition, step.name); + } + } + + loadStepDocumentation(connectorDefinition: ConnectorDefinition, stepName: string): void { + const stepState = this.getStepDocumentationState(stepName); + if (stepState && (stepState.status === 'loading' || stepState.status === 'success')) { + return; + } + + this.store.dispatch( + loadStepDocumentation({ + coordinates: { + group: connectorDefinition.group, + artifact: connectorDefinition.artifact, + version: connectorDefinition.version, + type: connectorDefinition.type + }, + stepName + }) + ); + } + + getStepDocumentationState(stepName: string): StepDocumentationState | undefined { + return this.connectorDefinitionState?.stepDocumentation[stepName]; + } + + isStepDocumentationLoading(stepName: string): boolean { + const state = this.getStepDocumentationState(stepName); + return state?.status === 'loading'; + } + + getStepDocumentation(stepName: string): string | undefined { + return this.getStepDocumentationState(stepName)?.documentation ?? undefined; + } + + getStepDocumentationError(stepName: string): string | undefined { + return this.getStepDocumentationState(stepName)?.error ?? undefined; + } + + ngOnDestroy(): void { + this.store.dispatch(resetConnectorDefinitionState()); + } +} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/feature/flow-designer.module.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/feature/flow-designer.module.ts index d68b3c9f7520..bb3e63033a5d 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/feature/flow-designer.module.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/feature/flow-designer.module.ts @@ -30,6 +30,7 @@ import { ParameterEffects } from '../state/parameter/parameter.effects'; import { QueueEffects } from '../state/queue/queue.effects'; import { BannerText } from '../../../ui/common/banner-text/banner-text.component'; import { FlowAnalysisEffects } from '../state/flow-analysis/flow-analysis.effects'; +import { ComponentTypeNamePipe } from '@nifi/shared'; @NgModule({ declarations: [FlowDesigner], @@ -49,6 +50,7 @@ import { FlowAnalysisEffects } from '../state/flow-analysis/flow-analysis.effect NgOptimizedImage, MatDialogModule, BannerText - ] + ], + providers: [ComponentTypeNamePipe] }) export class FlowDesignerModule {} diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/service/canvas-context-menu.service.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/service/canvas-context-menu.service.ts index ab7cdbe04b1d..2e8baea059e5 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/service/canvas-context-menu.service.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/service/canvas-context-menu.service.ts @@ -887,7 +887,8 @@ export class CanvasContextMenu implements ContextMenuDefinitionProvider { const selectionData = selection.datum(); this.store.dispatch( navigateToProvenanceForComponent({ - id: selectionData.id + id: selectionData.id, + componentType: selectionData.type }) ); } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.actions.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.actions.ts index a7fdc481555b..380a437f7eaf 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.actions.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.actions.ts @@ -121,6 +121,7 @@ import { } from '../../../../state/shared'; import { ErrorContext } from '../../../../state/error'; import { CopyResponseContext, CopyResponseEntity } from '../../../../state/copy'; +import { ComponentType } from '@nifi/shared'; const CANVAS_PREFIX = '[Canvas]'; @@ -614,7 +615,7 @@ export const renderConnectionsForComponent = createAction( export const navigateToProvenanceForComponent = createAction( `${CANVAS_PREFIX} Navigate To Provenance For Component`, - props<{ id: string }>() + props<{ id: string; componentType: ComponentType }>() ); export const replayLastProvenanceEvent = createAction( diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.spec.ts index ba287885d743..70cf7bcc7104 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.spec.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.spec.ts @@ -58,9 +58,10 @@ import { CopyPasteService } from '../../service/copy-paste.service'; import { CanvasView } from '../../service/canvas-view.service'; import { BirdseyeView } from '../../service/birdseye-view.service'; import { selectDisconnectionAcknowledged } from '../../../../state/cluster-summary/cluster-summary.selectors'; -import { ComponentType } from '@nifi/shared'; +import { ComponentType, ComponentTypeNamePipe } from '@nifi/shared'; import { ParameterContextService } from '../../../parameter-contexts/service/parameter-contexts.service'; import { HttpErrorResponse } from '@angular/common/http'; +import { provideRouter, Router } from '@angular/router'; import { ErrorHelper } from '../../../../service/error-helper.service'; import { selectConnectedStateChanged } from '../../../../state/cluster-summary/cluster-summary.selectors'; @@ -775,6 +776,8 @@ describe('FlowEffects', () => { imports: [], providers: [ FlowEffects, + ComponentTypeNamePipe, + provideRouter([]), provideMockActions(() => action$), provideMockStore({ selectors: [ @@ -1182,4 +1185,103 @@ describe('FlowEffects', () => { ); }); }); + + describe('navigateToProvenanceForComponent$', () => { + let router: Router; + + beforeEach(() => { + router = TestBed.inject(Router); + jest.spyOn(router, 'navigate').mockImplementation(() => Promise.resolve(true)); + store.overrideSelector(selectCurrentProcessGroupId, 'pg-123'); + store.refreshState(); + }); + + it('should navigate to provenance with Processor type in back navigation route and context', () => { + effects.navigateToProvenanceForComponent$.subscribe(); + + action$.next( + FlowActions.navigateToProvenanceForComponent({ + id: 'comp-1', + componentType: ComponentType.Processor + }) + ); + + expect(router.navigate).toHaveBeenCalledWith(['/provenance'], { + queryParams: { componentId: 'comp-1' }, + state: { + backNavigation: { + route: ['/process-groups', 'pg-123', ComponentType.Processor, 'comp-1'], + routeBoundary: ['/provenance'], + context: 'Processor' + } + } + }); + }); + + it('should navigate to provenance with Funnel type in back navigation route and context', () => { + effects.navigateToProvenanceForComponent$.subscribe(); + + action$.next( + FlowActions.navigateToProvenanceForComponent({ + id: 'funnel-1', + componentType: ComponentType.Funnel + }) + ); + + expect(router.navigate).toHaveBeenCalledWith(['/provenance'], { + queryParams: { componentId: 'funnel-1' }, + state: { + backNavigation: { + route: ['/process-groups', 'pg-123', ComponentType.Funnel, 'funnel-1'], + routeBoundary: ['/provenance'], + context: 'Funnel' + } + } + }); + }); + + it('should navigate to provenance with InputPort type in back navigation route and context', () => { + effects.navigateToProvenanceForComponent$.subscribe(); + + action$.next( + FlowActions.navigateToProvenanceForComponent({ + id: 'port-1', + componentType: ComponentType.InputPort + }) + ); + + expect(router.navigate).toHaveBeenCalledWith(['/provenance'], { + queryParams: { componentId: 'port-1' }, + state: { + backNavigation: { + route: ['/process-groups', 'pg-123', ComponentType.InputPort, 'port-1'], + routeBoundary: ['/provenance'], + context: 'Input Port' + } + } + }); + }); + + it('should navigate to provenance with OutputPort type in back navigation route and context', () => { + effects.navigateToProvenanceForComponent$.subscribe(); + + action$.next( + FlowActions.navigateToProvenanceForComponent({ + id: 'port-2', + componentType: ComponentType.OutputPort + }) + ); + + expect(router.navigate).toHaveBeenCalledWith(['/provenance'], { + queryParams: { componentId: 'port-2' }, + state: { + backNavigation: { + route: ['/process-groups', 'pg-123', ComponentType.OutputPort, 'port-2'], + routeBoundary: ['/provenance'], + context: 'Output Port' + } + } + }); + }); + }); }); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.ts index 658b630af5d3..79fba048ad0a 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/flow-designer/state/flow/flow.effects.ts @@ -128,6 +128,7 @@ import { EditProcessGroup } from '../../ui/canvas/items/process-group/edit-proce import { ControllerServiceService } from '../../service/controller-service.service'; import { ComponentType, + ComponentTypeNamePipe, isDefinedAndNotNull, LARGE_DIALOG, MEDIUM_DIALOG, @@ -200,6 +201,7 @@ export class FlowEffects { private parameterContextService = inject(ParameterContextService); private extensionTypesService = inject(ExtensionTypesService); private errorHelper = inject(ErrorHelper); + private componentTypeNamePipe = inject(ComponentTypeNamePipe); private copyPasteService = inject(CopyPasteService); private createProcessGroupDialogRef: MatDialogRef | undefined; @@ -2990,16 +2992,16 @@ export class FlowEffects { () => this.actions$.pipe( ofType(FlowActions.navigateToProvenanceForComponent), - map((action) => action.id), + map((action) => ({ id: action.id, componentType: action.componentType })), concatLatestFrom(() => this.store.select(selectCurrentProcessGroupId)), - tap(([componentId, processGroupId]) => { + tap(([{ id: componentId, componentType }, processGroupId]) => { this.router.navigate(['/provenance'], { queryParams: { componentId }, state: { backNavigation: { - route: ['/process-groups', processGroupId, ComponentType.Processor, componentId], + route: ['/process-groups', processGroupId, componentType, componentId], routeBoundary: ['/provenance'], - context: 'Processor' + context: this.componentTypeNamePipe.transform(componentType) } as BackNavigation } }); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/index.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/index.ts index 8cb87708e5c6..f8d95b4fba2a 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/index.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/index.ts @@ -41,6 +41,7 @@ export interface ProvenanceEventRequest { export interface GoToProvenanceEventSourceRequest { eventId?: number; componentId?: string; + componentType?: string; groupId?: string; clusterNodeId?: string; } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/provenance-event-listing.effects.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/provenance-event-listing.effects.ts index 46cb2e791526..9ff84a4d93af 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/provenance-event-listing.effects.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/state/provenance-event-listing/provenance-event-listing.effects.ts @@ -444,7 +444,9 @@ export class ProvenanceEventListingEffects { this.provenanceService.getProvenanceEvent(request.eventId, request.clusterNodeId).subscribe({ next: (response) => { const event: any = response.provenanceEvent; - this.router.navigate(this.getEventComponentLink(event.groupId, event.componentId)); + this.router.navigate( + this.getEventComponentLink(event.groupId, event.componentId, event.componentType) + ); }, error: (errorResponse: HttpErrorResponse) => { this.store.dispatch( @@ -455,7 +457,9 @@ export class ProvenanceEventListingEffects { } }); } else if (request.groupId && request.componentId) { - this.router.navigate(this.getEventComponentLink(request.groupId, request.componentId)); + this.router.navigate( + this.getEventComponentLink(request.groupId, request.componentId, request.componentType) + ); } }) ), @@ -495,15 +499,19 @@ export class ProvenanceEventListingEffects { { dispatch: false } ); - private getEventComponentLink(groupId: string, componentId: string): string[] { + private getEventComponentLink(groupId: string, componentId: string, componentType?: string): string[] { let link: string[]; if (groupId == componentId) { link = ['/process-groups', componentId]; - } else if (componentId === 'Connection' || componentId === 'Load Balanced Connection') { + } else if (componentType === 'Connection' || componentType === 'Load Balanced Connection') { link = ['/process-groups', groupId, 'Connection', componentId]; - } else if (componentId === 'Output Port') { + } else if (componentType === 'Output Port') { link = ['/process-groups', groupId, 'OutputPort', componentId]; + } else if (componentType === 'Input Port') { + link = ['/process-groups', groupId, 'InputPort', componentId]; + } else if (componentType === 'Funnel') { + link = ['/process-groups', groupId, 'Funnel', componentId]; } else { link = ['/process-groups', groupId, 'Processor', componentId]; } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/ui/provenance-event-listing/provenance-event-table/provenance-event-table.component.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/ui/provenance-event-listing/provenance-event-table/provenance-event-table.component.ts index e2df47bb3584..adb62661308e 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/ui/provenance-event-listing/provenance-event-table/provenance-event-table.component.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/provenance/ui/provenance-event-listing/provenance-event-table/provenance-event-table.component.ts @@ -338,12 +338,13 @@ export class ProvenanceEventTable implements AfterViewInit { return false; } - return !(event.componentId === 'Remote Output Port' || event.componentId === 'Remote Input Port'); + return !(event.componentType === 'Remote Output Port' || event.componentType === 'Remote Input Port'); } goToClicked(event: ProvenanceEventSummary): void { this.goToEventSource({ componentId: event.componentId, + componentType: event.componentType, groupId: event.groupId }); } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/flow-analysis-rules/flow-analysis-rule-table/flow-analysis-rule-table.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/flow-analysis-rules/flow-analysis-rule-table/flow-analysis-rule-table.component.spec.ts index dca5347d2eb5..953caea16845 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/flow-analysis-rules/flow-analysis-rule-table/flow-analysis-rule-table.component.spec.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/flow-analysis-rules/flow-analysis-rule-table/flow-analysis-rule-table.component.spec.ts @@ -134,6 +134,10 @@ describe('FlowAnalysisRuleTable', () => { canRead: true, canWrite: true }, + connectorsPermissions: { + canRead: true, + canWrite: true + }, componentRestrictionPermissions: [], canVersionFlows: true, logoutSupported: true diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/management-controller-services/management-controller-services.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/management-controller-services/management-controller-services.component.spec.ts index 55a5cf4db5ed..82f7012812fa 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/management-controller-services/management-controller-services.component.spec.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/management-controller-services/management-controller-services.component.spec.ts @@ -127,6 +127,10 @@ describe('ManagementControllerServices', () => { canRead: true, canWrite: true }, + connectorsPermissions: { + canRead: true, + canWrite: true + }, restrictedComponentsPermissions: { canRead: true, canWrite: true diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/parameter-providers/parameter-providers-table/parameter-providers-table.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/parameter-providers/parameter-providers-table/parameter-providers-table.component.spec.ts index 0133931981f8..76d51030a015 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/parameter-providers/parameter-providers-table/parameter-providers-table.component.spec.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/parameter-providers/parameter-providers-table/parameter-providers-table.component.spec.ts @@ -122,6 +122,10 @@ describe('ParameterProvidersTable', () => { canRead: true, canWrite: true }, + connectorsPermissions: { + canRead: true, + canWrite: true + }, restrictedComponentsPermissions: { canRead: true, canWrite: true diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/reporting-tasks/reporting-task-table/reporting-task-table.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/reporting-tasks/reporting-task-table/reporting-task-table.component.spec.ts index 9008c53a15c1..cc79f9c2283d 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/reporting-tasks/reporting-task-table/reporting-task-table.component.spec.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/settings/ui/reporting-tasks/reporting-task-table/reporting-task-table.component.spec.ts @@ -138,6 +138,10 @@ describe('ReportingTaskTable', () => { controllerPermissions: { canRead: true, canWrite: true + }, + connectorsPermissions: { + canRead: true, + canWrite: true } }; } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/users/ui/user-listing/user-table/user-table.component.spec.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/users/ui/user-listing/user-table/user-table.component.spec.ts index c4258373bbf4..e69bf072f9de 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/users/ui/user-listing/user-table/user-table.component.spec.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/pages/users/ui/user-listing/user-table/user-table.component.spec.ts @@ -61,6 +61,10 @@ describe('UserTable', () => { canRead: true, canWrite: true }, + connectorsPermissions: { + canRead: false, + canWrite: false + }, restrictedComponentsPermissions: { canRead: false, canWrite: true diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/service/extension-types.service.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/service/extension-types.service.ts index 477b1b6510cf..f76675842516 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/service/extension-types.service.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/service/extension-types.service.ts @@ -99,4 +99,8 @@ export class ExtensionTypesService { getParameterProviderTypes(): Observable { return this.httpClient.get(`${ExtensionTypesService.API}/flow/parameter-provider-types`); } + + getConnectorTypes(): Observable { + return this.httpClient.get(`${ExtensionTypesService.API}/flow/connector-types`); + } } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/current-user.reducer.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/current-user.reducer.ts index a1b57ea94fd3..3825549351f3 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/current-user.reducer.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/current-user.reducer.ts @@ -34,6 +34,7 @@ export const initialState: CurrentUserState = { controllerPermissions: NO_PERMISSIONS, countersPermissions: NO_PERMISSIONS, parameterContextPermissions: NO_PERMISSIONS, + connectorsPermissions: NO_PERMISSIONS, policiesPermissions: NO_PERMISSIONS, provenancePermissions: NO_PERMISSIONS, restrictedComponentsPermissions: NO_PERMISSIONS, diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/index.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/index.ts index 4dfd972c66d6..2f05556c7c38 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/index.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/current-user/index.ts @@ -41,6 +41,7 @@ export interface CurrentUser { policiesPermissions: Permissions; systemPermissions: Permissions; parameterContextPermissions: Permissions; + connectorsPermissions: Permissions; restrictedComponentsPermissions: Permissions; componentRestrictionPermissions: ComponentRestrictionPermission[]; } diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.effects.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.effects.ts index 256d15e192ea..98d411850acb 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.effects.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.effects.ts @@ -136,7 +136,8 @@ export class ExtensionTypesEffects { this.extensionTypesService.getReportingTaskTypes(), this.extensionTypesService.getRegistryClientTypes(), this.extensionTypesService.getParameterProviderTypes(), - this.extensionTypesService.getFlowAnalysisRuleTypes() + this.extensionTypesService.getFlowAnalysisRuleTypes(), + this.extensionTypesService.getConnectorTypes() ]).pipe( map( ([ @@ -145,7 +146,8 @@ export class ExtensionTypesEffects { reportingTaskTypes, registryClientTypes, parameterProviderTypes, - flowAnalysisRuleTypes + flowAnalysisRuleTypes, + connectorTypes ]) => ExtensionTypesActions.loadExtensionTypesForDocumentationSuccess({ response: { @@ -154,7 +156,8 @@ export class ExtensionTypesEffects { reportingTaskTypes: reportingTaskTypes.reportingTaskTypes, registryClientTypes: registryClientTypes.flowRegistryClientTypes, parameterProviderTypes: parameterProviderTypes.parameterProviderTypes, - flowAnalysisRuleTypes: flowAnalysisRuleTypes.flowAnalysisRuleTypes + flowAnalysisRuleTypes: flowAnalysisRuleTypes.flowAnalysisRuleTypes, + connectorTypes: connectorTypes.connectorTypes } }) ), diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.reducer.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.reducer.ts index a06944ca42b3..3ac1f8c91824 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.reducer.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.reducer.ts @@ -37,6 +37,7 @@ export const initialExtensionsTypesState: ExtensionTypesState = { registryClientTypes: [], flowAnalysisRuleTypes: [], parameterProviderTypes: [], + connectorTypes: [], status: 'pending' }; @@ -56,6 +57,7 @@ export const extensionTypesReducer = createReducer( registryClientTypes: [], parameterProviderTypes: [], flowAnalysisRuleTypes: [], + connectorTypes: [], status: 'loading' as const }) ), @@ -92,6 +94,7 @@ export const extensionTypesReducer = createReducer( registryClientTypes: response.registryClientTypes, parameterProviderTypes: response.parameterProviderTypes, flowAnalysisRuleTypes: response.flowAnalysisRuleTypes, + connectorTypes: response.connectorTypes, status: 'success' as const })), on(extensionTypesApiError, (state) => ({ diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.selectors.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.selectors.ts index 65cc3e4f57c9..059a4da62e68 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.selectors.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/extension-types.selectors.ts @@ -61,6 +61,11 @@ export const selectParameterProviderTypes = createSelector( (state: ExtensionTypesState) => state.parameterProviderTypes ); +export const selectConnectorTypes = createSelector( + selectExtensionTypesState, + (state: ExtensionTypesState) => state.connectorTypes +); + export const selectTypesToIdentifyComponentRestrictions = createSelector( selectExtensionTypesState, (state: ExtensionTypesState) => { @@ -139,6 +144,7 @@ export const selectExtensionFromTypes = (extensionTypes: string[]) => reportingTaskTypes: state.reportingTaskTypes.filter(typeFilter), registryClientTypes: state.registryClientTypes.filter(typeFilter), parameterProviderTypes: state.parameterProviderTypes.filter(typeFilter), - flowAnalysisRuleTypes: state.flowAnalysisRuleTypes.filter(typeFilter) + flowAnalysisRuleTypes: state.flowAnalysisRuleTypes.filter(typeFilter), + connectorTypes: state.connectorTypes.filter(typeFilter) } as LoadExtensionTypesForDocumentationResponse; }); diff --git a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/index.ts b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/index.ts index 05d2d351d090..249772cb9422 100644 --- a/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/index.ts +++ b/nifi-frontend/src/main/frontend/apps/nifi/src/app/state/extension-types/index.ts @@ -50,6 +50,7 @@ export interface LoadExtensionTypesForDocumentationResponse { registryClientTypes: DocumentedType[]; flowAnalysisRuleTypes: DocumentedType[]; parameterProviderTypes: DocumentedType[]; + connectorTypes: DocumentedType[]; } export interface ExtensionTypesState { @@ -60,5 +61,6 @@ export interface ExtensionTypesState { registryClientTypes: DocumentedType[]; flowAnalysisRuleTypes: DocumentedType[]; parameterProviderTypes: DocumentedType[]; + connectorTypes: DocumentedType[]; status: ExtensionTypesLoadingStatus; } diff --git a/nifi-frontend/src/main/frontend/libs/shared/src/services/nifi-common.service.ts b/nifi-frontend/src/main/frontend/libs/shared/src/services/nifi-common.service.ts index bb6e285c85fd..75d5973b5cb9 100644 --- a/nifi-frontend/src/main/frontend/libs/shared/src/services/nifi-common.service.ts +++ b/nifi-frontend/src/main/frontend/libs/shared/src/services/nifi-common.service.ts @@ -63,6 +63,11 @@ export class NiFiCommon { value: 'parameter-contexts', description: 'Allows users to view/modify Parameter Contexts' }, + { + text: 'access connectors', + value: 'connectors', + description: 'Allows users to view/modify Connectors' + }, { text: 'query provenance', value: 'provenance', diff --git a/nifi-frontend/src/main/frontend/libs/shared/src/types/index.ts b/nifi-frontend/src/main/frontend/libs/shared/src/types/index.ts index 5f1e21ab133e..25e922995dcc 100644 --- a/nifi-frontend/src/main/frontend/libs/shared/src/types/index.ts +++ b/nifi-frontend/src/main/frontend/libs/shared/src/types/index.ts @@ -57,6 +57,7 @@ export enum ComponentType { FlowAnalysisRule = 'FlowAnalysisRule', ParameterProvider = 'ParameterProvider', FlowRegistryClient = 'FlowRegistryClient', + Connector = 'Connector', Flow = 'Flow' } diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConfigurationStep.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConfigurationStep.java new file mode 100644 index 000000000000..457e6d8fe8b1 --- /dev/null +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConfigurationStep.java @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.extension.manifest; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlElement; +import jakarta.xml.bind.annotation.XmlElementWrapper; + +import java.util.List; + +/** + * Represents a configuration step for a Connector. + */ +@XmlAccessorType(XmlAccessType.FIELD) +public class ConfigurationStep { + + private String name; + private String description; + + @XmlElementWrapper + @XmlElement(name = "stepDependency") + private List stepDependencies; + + @XmlElementWrapper + @XmlElement(name = "propertyGroup") + private List propertyGroups; + + @Schema(description = "The name of the configuration step") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Schema(description = "The description of the configuration step") + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Schema(description = "The dependencies that this step has on other steps") + public List getStepDependencies() { + return stepDependencies; + } + + public void setStepDependencies(List stepDependencies) { + this.stepDependencies = stepDependencies; + } + + @Schema(description = "The property groups in this configuration step") + public List getPropertyGroups() { + return propertyGroups; + } + + public void setPropertyGroups(List propertyGroups) { + this.propertyGroups = propertyGroups; + } +} + diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConfigurationStepDependency.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConfigurationStepDependency.java new file mode 100644 index 000000000000..ac4381b3e075 --- /dev/null +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConfigurationStepDependency.java @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.extension.manifest; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlElement; +import jakarta.xml.bind.annotation.XmlElementWrapper; + +import java.util.List; + +/** + * Represents a dependency that a configuration step has on another step's property. + */ +@XmlAccessorType(XmlAccessType.FIELD) +public class ConfigurationStepDependency { + + private String stepName; + private String propertyName; + + @XmlElementWrapper + @XmlElement(name = "dependentValue") + private List dependentValues; + + @Schema(description = "The name of the step that this step depends on") + public String getStepName() { + return stepName; + } + + public void setStepName(String stepName) { + this.stepName = stepName; + } + + @Schema(description = "The name of the property within the step that this step depends on") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(String propertyName) { + this.propertyName = propertyName; + } + + @Schema(description = "The values of the dependent property that enable this step") + public List getDependentValues() { + return dependentValues; + } + + public void setDependentValues(List dependentValues) { + this.dependentValues = dependentValues; + } +} + diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorProperty.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorProperty.java new file mode 100644 index 000000000000..812a45a88090 --- /dev/null +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorProperty.java @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.extension.manifest; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlElement; +import jakarta.xml.bind.annotation.XmlElementWrapper; + +import java.util.List; + +/** + * Represents a property descriptor for a Connector. + */ +@XmlAccessorType(XmlAccessType.FIELD) +public class ConnectorProperty { + + private String name; + private String description; + private String defaultValue; + private boolean required; + private ConnectorPropertyType propertyType; + private boolean allowableValuesFetchable; + + @XmlElementWrapper + @XmlElement(name = "allowableValue") + private List allowableValues; + + @XmlElementWrapper + @XmlElement(name = "dependency") + private List dependencies; + + @Schema(description = "The name of the property") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Schema(description = "The description of the property") + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Schema(description = "The default value of the property") + public String getDefaultValue() { + return defaultValue; + } + + public void setDefaultValue(String defaultValue) { + this.defaultValue = defaultValue; + } + + @Schema(description = "Whether or not the property is required") + public boolean isRequired() { + return required; + } + + public void setRequired(boolean required) { + this.required = required; + } + + @Schema(description = "The type of the property") + public ConnectorPropertyType getPropertyType() { + return propertyType; + } + + public void setPropertyType(ConnectorPropertyType propertyType) { + this.propertyType = propertyType; + } + + @Schema(description = "Whether or not the allowable values can be fetched dynamically") + public boolean isAllowableValuesFetchable() { + return allowableValuesFetchable; + } + + public void setAllowableValuesFetchable(boolean allowableValuesFetchable) { + this.allowableValuesFetchable = allowableValuesFetchable; + } + + @Schema(description = "The allowable values for this property") + public List getAllowableValues() { + return allowableValues; + } + + public void setAllowableValues(List allowableValues) { + this.allowableValues = allowableValues; + } + + @Schema(description = "The properties that this property depends on") + public List getDependencies() { + return dependencies; + } + + public void setDependencies(List dependencies) { + this.dependencies = dependencies; + } +} + diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyDependency.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyDependency.java new file mode 100644 index 000000000000..dc1fb67a6b84 --- /dev/null +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyDependency.java @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.extension.manifest; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlElement; +import jakarta.xml.bind.annotation.XmlElementWrapper; + +import java.util.List; + +/** + * Represents a dependency that a connector property has on another property. + */ +@XmlAccessorType(XmlAccessType.FIELD) +public class ConnectorPropertyDependency { + + private String propertyName; + + @XmlElementWrapper + @XmlElement(name = "dependentValue") + private List dependentValues; + + @Schema(description = "The name of the property that this property depends on") + public String getPropertyName() { + return propertyName; + } + + public void setPropertyName(String propertyName) { + this.propertyName = propertyName; + } + + @Schema(description = "The values of the dependent property that enable this property") + public List getDependentValues() { + return dependentValues; + } + + public void setDependentValues(List dependentValues) { + this.dependentValues = dependentValues; + } +} + diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyGroup.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyGroup.java new file mode 100644 index 000000000000..e6d411517c32 --- /dev/null +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyGroup.java @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.extension.manifest; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlElement; +import jakarta.xml.bind.annotation.XmlElementWrapper; + +import java.util.List; + +/** + * Represents a group of properties within a configuration step. + */ +@XmlAccessorType(XmlAccessType.FIELD) +public class ConnectorPropertyGroup { + + private String name; + private String description; + + @XmlElementWrapper + @XmlElement(name = "property") + private List properties; + + @Schema(description = "The name of the property group") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Schema(description = "The description of the property group") + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Schema(description = "The properties in this group") + public List getProperties() { + return properties; + } + + public void setProperties(List properties) { + this.properties = properties; + } +} + diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyType.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyType.java new file mode 100644 index 000000000000..07ced332c753 --- /dev/null +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ConnectorPropertyType.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.extension.manifest; + +/** + * Possible types for connector properties. + */ +public enum ConnectorPropertyType { + + STRING, + + INTEGER, + + BOOLEAN, + + FLOAT, + + DOUBLE, + + STRING_LIST, + + SECRET, + + ASSET, + + ASSET_LIST; + +} + diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/Extension.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/Extension.java index 53e739cbadf9..56ba1d4928e4 100644 --- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/Extension.java +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/Extension.java @@ -107,6 +107,11 @@ public class Extension { @XmlElement(name = "multiProcessorUseCase") private List multiProcessorUseCases; + // Connector-specific fields + @XmlElementWrapper + @XmlElement(name = "configurationStep") + private List configurationSteps; + @Schema(description = "The name of the extension") public String getName() { return name; @@ -358,6 +363,15 @@ public void setMultiProcessorUseCases(final List multiPro this.multiProcessorUseCases = multiProcessorUseCases; } + @Schema(description = "The configuration steps for a Connector extension") + public List getConfigurationSteps() { + return configurationSteps; + } + + public void setConfigurationSteps(final List configurationSteps) { + this.configurationSteps = configurationSteps; + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java index 0413e260ed12..a72511dabd09 100644 --- a/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java +++ b/nifi-manifest/nifi-extension-manifest-model/src/main/java/org/apache/nifi/extension/manifest/ExtensionType.java @@ -31,6 +31,8 @@ public enum ExtensionType { PARAMETER_PROVIDER, - FLOW_REGISTRY_CLIENT; + FLOW_REGISTRY_CLIENT, + + CONNECTOR; } diff --git a/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/ComponentManifestBuilder.java b/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/ComponentManifestBuilder.java index 9eb3f697ffb0..4ced8551411e 100644 --- a/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/ComponentManifestBuilder.java +++ b/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/ComponentManifestBuilder.java @@ -17,6 +17,7 @@ package org.apache.nifi.runtime.manifest; import org.apache.nifi.c2.protocol.component.api.ComponentManifest; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.FlowAnalysisRuleDefinition; import org.apache.nifi.c2.protocol.component.api.FlowRegistryClientDefinition; @@ -65,6 +66,12 @@ public interface ComponentManifestBuilder { */ ComponentManifestBuilder addFlowRegistryClient(FlowRegistryClientDefinition flowRegistryClientDefinition); + /** + * @param connectorDefinition a connector definition to add + * @return the builder + */ + ComponentManifestBuilder addConnector(ConnectorDefinition connectorDefinition); + /** * @return a component manifest containing all the added definitions */ diff --git a/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardComponentManifestBuilder.java b/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardComponentManifestBuilder.java index b6b67181bc6d..712aab218768 100644 --- a/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardComponentManifestBuilder.java +++ b/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardComponentManifestBuilder.java @@ -17,6 +17,7 @@ package org.apache.nifi.runtime.manifest.impl; import org.apache.nifi.c2.protocol.component.api.ComponentManifest; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.FlowAnalysisRuleDefinition; import org.apache.nifi.c2.protocol.component.api.FlowRegistryClientDefinition; @@ -39,6 +40,7 @@ public class StandardComponentManifestBuilder implements ComponentManifestBuilde private final List parameterProviders = new ArrayList<>(); private final List flowAnalysisRules = new ArrayList<>(); private final List flowRegistryClients = new ArrayList<>(); + private final List connectors = new ArrayList<>(); @Override public ComponentManifestBuilder addProcessor(final ProcessorDefinition processorDefinition) { @@ -94,6 +96,15 @@ public ComponentManifestBuilder addFlowRegistryClient(final FlowRegistryClientDe return this; } + @Override + public ComponentManifestBuilder addConnector(final ConnectorDefinition connectorDefinition) { + if (connectorDefinition == null) { + throw new IllegalArgumentException("Connector definition cannot be null"); + } + connectors.add(connectorDefinition); + return this; + } + @Override public ComponentManifest build() { final ComponentManifest componentManifest = new ComponentManifest(); @@ -103,6 +114,7 @@ public ComponentManifest build() { componentManifest.setParameterProviders(new ArrayList<>(parameterProviders)); componentManifest.setFlowAnalysisRules(new ArrayList<>(flowAnalysisRules)); componentManifest.setFlowRegistryClients(new ArrayList<>(flowRegistryClients)); + componentManifest.setConnectors(new ArrayList<>(connectors)); return componentManifest; } diff --git a/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilder.java b/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilder.java index a080378e5eae..91318bac00a6 100644 --- a/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilder.java +++ b/nifi-manifest/nifi-runtime-manifest-core/src/main/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilder.java @@ -20,6 +20,13 @@ import org.apache.nifi.c2.protocol.component.api.BuildInfo; import org.apache.nifi.c2.protocol.component.api.Bundle; import org.apache.nifi.c2.protocol.component.api.ConfigurableComponentDefinition; +import org.apache.nifi.c2.protocol.component.api.ConfigurationStep; +import org.apache.nifi.c2.protocol.component.api.ConfigurationStepDependency; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDependency; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDescriptor; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyGroup; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyType; import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition; import org.apache.nifi.c2.protocol.component.api.DefinedType; import org.apache.nifi.c2.protocol.component.api.ExtensionComponent; @@ -46,6 +53,7 @@ import org.apache.nifi.expression.ExpressionLanguageScope; import org.apache.nifi.extension.manifest.AllowableValue; import org.apache.nifi.extension.manifest.Attribute; +import org.apache.nifi.extension.manifest.ConnectorProperty; import org.apache.nifi.extension.manifest.DefaultSchedule; import org.apache.nifi.extension.manifest.DefaultSettings; import org.apache.nifi.extension.manifest.Dependency; @@ -226,6 +234,9 @@ private void addExtension(final ExtensionManifest extensionManifest, final Exten case FLOW_REGISTRY_CLIENT: addFlowRegistryClientDefinition(extensionManifest, extension, additionalDetails, componentManifestBuilder); break; + case CONNECTOR: + addConnectorDefinition(extensionManifest, extension, additionalDetails, componentManifestBuilder); + break; } } @@ -410,6 +421,121 @@ private void addFlowRegistryClientDefinition(final ExtensionManifest extensionMa componentManifestBuilder.addFlowRegistryClient(flowRegistryClientDefinition); } + private void addConnectorDefinition(final ExtensionManifest extensionManifest, final Extension extension, final String additionalDetails, + final ComponentManifestBuilder componentManifestBuilder) { + final ConnectorDefinition connectorDefinition = new ConnectorDefinition(); + populateDefinedType(extensionManifest, extension, connectorDefinition); + populateExtensionComponent(extensionManifest, extension, additionalDetails, connectorDefinition); + + // Populate configuration steps + final List manifestSteps = extension.getConfigurationSteps(); + if (isNotEmpty(manifestSteps)) { + connectorDefinition.setConfigurationSteps( + manifestSteps.stream() + .map(this::getConfigurationStep) + .collect(Collectors.toList()) + ); + } + + componentManifestBuilder.addConnector(connectorDefinition); + } + + private ConfigurationStep getConfigurationStep(final org.apache.nifi.extension.manifest.ConfigurationStep manifestStep) { + final ConfigurationStep step = new ConfigurationStep(); + step.setName(manifestStep.getName()); + step.setDescription(manifestStep.getDescription()); + + // Convert step dependencies + final List manifestDeps = manifestStep.getStepDependencies(); + if (isNotEmpty(manifestDeps)) { + step.setStepDependencies( + manifestDeps.stream() + .map(this::getConfigurationStepDependency) + .collect(Collectors.toList()) + ); + } + + // Convert property groups + final List manifestGroups = manifestStep.getPropertyGroups(); + if (isNotEmpty(manifestGroups)) { + step.setPropertyGroups( + manifestGroups.stream() + .map(this::getConnectorPropertyGroup) + .collect(Collectors.toList()) + ); + } + + return step; + } + + private ConfigurationStepDependency getConfigurationStepDependency( + final org.apache.nifi.extension.manifest.ConfigurationStepDependency manifestDep) { + final ConfigurationStepDependency dep = new ConfigurationStepDependency(); + dep.setStepName(manifestDep.getStepName()); + dep.setPropertyName(manifestDep.getPropertyName()); + dep.setDependentValues(manifestDep.getDependentValues()); + return dep; + } + + private ConnectorPropertyGroup getConnectorPropertyGroup( + final org.apache.nifi.extension.manifest.ConnectorPropertyGroup manifestGroup) { + final ConnectorPropertyGroup group = new ConnectorPropertyGroup(); + group.setName(manifestGroup.getName()); + group.setDescription(manifestGroup.getDescription()); + + final List manifestProps = manifestGroup.getProperties(); + if (isNotEmpty(manifestProps)) { + group.setProperties( + manifestProps.stream() + .map(this::getConnectorPropertyDescriptor) + .collect(Collectors.toList()) + ); + } + + return group; + } + + private ConnectorPropertyDescriptor getConnectorPropertyDescriptor(final ConnectorProperty manifestProp) { + final ConnectorPropertyDescriptor prop = new ConnectorPropertyDescriptor(); + prop.setName(manifestProp.getName()); + prop.setDescription(manifestProp.getDescription()); + prop.setDefaultValue(manifestProp.getDefaultValue()); + prop.setRequired(manifestProp.isRequired()); + prop.setAllowableValuesFetchable(manifestProp.isAllowableValuesFetchable()); + + // Convert property type + final org.apache.nifi.extension.manifest.ConnectorPropertyType manifestType = manifestProp.getPropertyType(); + if (manifestType != null) { + prop.setPropertyType(ConnectorPropertyType.valueOf(manifestType.name())); + } + + // Convert allowable values + final List manifestValues = manifestProp.getAllowableValues(); + if (isNotEmpty(manifestValues)) { + prop.setAllowableValues(getPropertyAllowableValues(manifestValues)); + } + + // Convert property dependencies + final List manifestDeps = manifestProp.getDependencies(); + if (isNotEmpty(manifestDeps)) { + prop.setDependencies( + manifestDeps.stream() + .map(this::getConnectorPropertyDependency) + .collect(Collectors.toList()) + ); + } + + return prop; + } + + private ConnectorPropertyDependency getConnectorPropertyDependency( + final org.apache.nifi.extension.manifest.ConnectorPropertyDependency manifestDep) { + final ConnectorPropertyDependency dep = new ConnectorPropertyDependency(); + dep.setPropertyName(manifestDep.getPropertyName()); + dep.setDependentValues(manifestDep.getDependentValues()); + return dep; + } + private void addFlowAnalysisRuleDefinition(final ExtensionManifest extensionManifest, final Extension extension, final String additionalDetails, final ComponentManifestBuilder componentManifestBuilder) { final FlowAnalysisRuleDefinition flowAnalysisRuleDefinition = new FlowAnalysisRuleDefinition(); diff --git a/nifi-manifest/nifi-runtime-manifest-core/src/test/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilderConnectorTest.java b/nifi-manifest/nifi-runtime-manifest-core/src/test/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilderConnectorTest.java new file mode 100644 index 000000000000..8585dc479574 --- /dev/null +++ b/nifi-manifest/nifi-runtime-manifest-core/src/test/java/org/apache/nifi/runtime/manifest/impl/StandardRuntimeManifestBuilderConnectorTest.java @@ -0,0 +1,349 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.runtime.manifest.impl; + +import org.apache.nifi.c2.protocol.component.api.ConfigurationStep; +import org.apache.nifi.c2.protocol.component.api.ConfigurationStepDependency; +import org.apache.nifi.c2.protocol.component.api.ConnectorDefinition; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDescriptor; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyGroup; +import org.apache.nifi.c2.protocol.component.api.ConnectorPropertyType; +import org.apache.nifi.c2.protocol.component.api.RuntimeManifest; +import org.apache.nifi.extension.manifest.AllowableValue; +import org.apache.nifi.extension.manifest.ConnectorProperty; +import org.apache.nifi.extension.manifest.ConnectorPropertyDependency; +import org.apache.nifi.extension.manifest.Extension; +import org.apache.nifi.extension.manifest.ExtensionManifest; +import org.apache.nifi.extension.manifest.ExtensionType; +import org.apache.nifi.extension.manifest.ParentNar; +import org.apache.nifi.runtime.manifest.ExtensionManifestContainer; +import org.junit.jupiter.api.Test; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class StandardRuntimeManifestBuilderConnectorTest { + + private static final String TEST_CONNECTOR_TYPE = "org.apache.nifi.connectors.TestConnector"; + private static final String TEST_GROUP = "org.apache.nifi"; + private static final String TEST_ARTIFACT = "nifi-test-nar"; + private static final String TEST_VERSION = "2.0.0"; + private static final String STEP_NAME = "Connection Settings"; + private static final String STEP_DESCRIPTION = "Configure connection parameters"; + private static final String GROUP_NAME = "Authentication"; + private static final String GROUP_DESCRIPTION = "Authentication settings"; + private static final String PROPERTY_NAME = "username"; + private static final String PROPERTY_DESCRIPTION = "The username for authentication"; + private static final String PROPERTY_DEFAULT = "admin"; + + @Test + void testConnectorDefinitionIsBuiltFromExtensionManifest() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtension(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + assertNotNull(runtimeManifest); + assertNotNull(runtimeManifest.getBundles()); + assertEquals(1, runtimeManifest.getBundles().size()); + + final List connectors = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors(); + assertNotNull(connectors); + assertEquals(1, connectors.size()); + + final ConnectorDefinition connector = connectors.get(0); + assertEquals(TEST_CONNECTOR_TYPE, connector.getType()); + } + + @Test + void testConfigurationStepsAreConverted() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtensionWithConfigurationSteps(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + final ConnectorDefinition connector = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors().get(0); + assertNotNull(connector.getConfigurationSteps()); + assertEquals(1, connector.getConfigurationSteps().size()); + + final ConfigurationStep step = connector.getConfigurationSteps().get(0); + assertEquals(STEP_NAME, step.getName()); + assertEquals(STEP_DESCRIPTION, step.getDescription()); + } + + @Test + void testConfigurationStepDependenciesAreConverted() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtensionWithStepDependencies(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + final ConfigurationStep step = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors().get(0).getConfigurationSteps().get(0); + assertNotNull(step.getStepDependencies()); + assertEquals(1, step.getStepDependencies().size()); + + final ConfigurationStepDependency dependency = step.getStepDependencies().get(0); + assertEquals("previousStep", dependency.getStepName()); + assertEquals("connectionType", dependency.getPropertyName()); + assertEquals(List.of("advanced"), dependency.getDependentValues()); + } + + @Test + void testPropertyGroupsAreConverted() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtensionWithPropertyGroups(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + final ConfigurationStep step = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors().get(0).getConfigurationSteps().get(0); + assertNotNull(step.getPropertyGroups()); + assertEquals(1, step.getPropertyGroups().size()); + + final ConnectorPropertyGroup group = step.getPropertyGroups().get(0); + assertEquals(GROUP_NAME, group.getName()); + assertEquals(GROUP_DESCRIPTION, group.getDescription()); + } + + @Test + void testConnectorPropertiesAreConverted() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtensionWithProperties(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + final ConnectorPropertyGroup group = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors().get(0) + .getConfigurationSteps().get(0).getPropertyGroups().get(0); + assertNotNull(group.getProperties()); + assertEquals(1, group.getProperties().size()); + + final ConnectorPropertyDescriptor property = group.getProperties().get(0); + assertEquals(PROPERTY_NAME, property.getName()); + assertEquals(PROPERTY_DESCRIPTION, property.getDescription()); + assertEquals(PROPERTY_DEFAULT, property.getDefaultValue()); + assertTrue(property.isRequired()); + assertFalse(property.isAllowableValuesFetchable()); + assertEquals(ConnectorPropertyType.STRING, property.getPropertyType()); + } + + @Test + void testConnectorPropertyAllowableValuesAreConverted() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtensionWithAllowableValues(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + final ConnectorPropertyDescriptor property = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors().get(0) + .getConfigurationSteps().get(0).getPropertyGroups().get(0).getProperties().get(0); + assertNotNull(property.getAllowableValues()); + assertEquals(2, property.getAllowableValues().size()); + assertEquals("option1", property.getAllowableValues().get(0).getValue()); + assertEquals("Option 1", property.getAllowableValues().get(0).getDisplayName()); + } + + @Test + void testConnectorPropertyDependenciesAreConverted() { + final ExtensionManifest extensionManifest = createExtensionManifest(); + final Extension connectorExtension = createConnectorExtensionWithPropertyDependencies(); + extensionManifest.setExtensions(List.of(connectorExtension)); + + final ExtensionManifestContainer container = new ExtensionManifestContainer(extensionManifest, null); + final RuntimeManifest runtimeManifest = new StandardRuntimeManifestBuilder() + .addBundle(container) + .build(); + + final ConnectorPropertyDescriptor property = runtimeManifest.getBundles().get(0).getComponentManifest().getConnectors().get(0) + .getConfigurationSteps().get(0).getPropertyGroups().get(0).getProperties().get(0); + assertNotNull(property.getDependencies()); + assertEquals(1, property.getDependencies().size()); + assertEquals("authType", property.getDependencies().get(0).getPropertyName()); + assertEquals(List.of("basic", "oauth"), property.getDependencies().get(0).getDependentValues()); + } + + private ExtensionManifest createExtensionManifest() { + final ExtensionManifest manifest = new ExtensionManifest(); + manifest.setGroupId(TEST_GROUP); + manifest.setArtifactId(TEST_ARTIFACT); + manifest.setVersion(TEST_VERSION); + + final ParentNar parentNar = new ParentNar(); + parentNar.setGroupId("org.apache.nifi"); + parentNar.setArtifactId("nifi-standard-services-api-nar"); + parentNar.setVersion(TEST_VERSION); + manifest.setParentNar(parentNar); + + return manifest; + } + + private Extension createConnectorExtension() { + final Extension extension = new Extension(); + extension.setName(TEST_CONNECTOR_TYPE); + extension.setType(ExtensionType.CONNECTOR); + return extension; + } + + private Extension createConnectorExtensionWithConfigurationSteps() { + final Extension extension = createConnectorExtension(); + + final org.apache.nifi.extension.manifest.ConfigurationStep step = new org.apache.nifi.extension.manifest.ConfigurationStep(); + step.setName(STEP_NAME); + step.setDescription(STEP_DESCRIPTION); + extension.setConfigurationSteps(List.of(step)); + + return extension; + } + + private Extension createConnectorExtensionWithStepDependencies() { + final Extension extension = createConnectorExtension(); + + final org.apache.nifi.extension.manifest.ConfigurationStepDependency dependency = new org.apache.nifi.extension.manifest.ConfigurationStepDependency(); + dependency.setStepName("previousStep"); + dependency.setPropertyName("connectionType"); + dependency.setDependentValues(List.of("advanced")); + + final org.apache.nifi.extension.manifest.ConfigurationStep step = new org.apache.nifi.extension.manifest.ConfigurationStep(); + step.setName(STEP_NAME); + step.setDescription(STEP_DESCRIPTION); + step.setStepDependencies(List.of(dependency)); + extension.setConfigurationSteps(List.of(step)); + + return extension; + } + + private Extension createConnectorExtensionWithPropertyGroups() { + final Extension extension = createConnectorExtension(); + + final org.apache.nifi.extension.manifest.ConnectorPropertyGroup group = new org.apache.nifi.extension.manifest.ConnectorPropertyGroup(); + group.setName(GROUP_NAME); + group.setDescription(GROUP_DESCRIPTION); + + final org.apache.nifi.extension.manifest.ConfigurationStep step = new org.apache.nifi.extension.manifest.ConfigurationStep(); + step.setName(STEP_NAME); + step.setDescription(STEP_DESCRIPTION); + step.setPropertyGroups(List.of(group)); + extension.setConfigurationSteps(List.of(step)); + + return extension; + } + + private Extension createConnectorExtensionWithProperties() { + final Extension extension = createConnectorExtension(); + + final ConnectorProperty property = new ConnectorProperty(); + property.setName(PROPERTY_NAME); + property.setDescription(PROPERTY_DESCRIPTION); + property.setDefaultValue(PROPERTY_DEFAULT); + property.setRequired(true); + property.setAllowableValuesFetchable(false); + property.setPropertyType(org.apache.nifi.extension.manifest.ConnectorPropertyType.STRING); + + final org.apache.nifi.extension.manifest.ConnectorPropertyGroup group = new org.apache.nifi.extension.manifest.ConnectorPropertyGroup(); + group.setName(GROUP_NAME); + group.setDescription(GROUP_DESCRIPTION); + group.setProperties(List.of(property)); + + final org.apache.nifi.extension.manifest.ConfigurationStep step = new org.apache.nifi.extension.manifest.ConfigurationStep(); + step.setName(STEP_NAME); + step.setDescription(STEP_DESCRIPTION); + step.setPropertyGroups(List.of(group)); + extension.setConfigurationSteps(List.of(step)); + + return extension; + } + + private Extension createConnectorExtensionWithAllowableValues() { + final Extension extension = createConnectorExtension(); + + final AllowableValue value1 = new AllowableValue(); + value1.setValue("option1"); + value1.setDisplayName("Option 1"); + value1.setDescription("First option"); + + final AllowableValue value2 = new AllowableValue(); + value2.setValue("option2"); + value2.setDisplayName("Option 2"); + value2.setDescription("Second option"); + + final ConnectorProperty property = new ConnectorProperty(); + property.setName(PROPERTY_NAME); + property.setDescription(PROPERTY_DESCRIPTION); + property.setAllowableValues(List.of(value1, value2)); + + final org.apache.nifi.extension.manifest.ConnectorPropertyGroup group = new org.apache.nifi.extension.manifest.ConnectorPropertyGroup(); + group.setName(GROUP_NAME); + group.setProperties(List.of(property)); + + final org.apache.nifi.extension.manifest.ConfigurationStep step = new org.apache.nifi.extension.manifest.ConfigurationStep(); + step.setName(STEP_NAME); + step.setPropertyGroups(List.of(group)); + extension.setConfigurationSteps(List.of(step)); + + return extension; + } + + private Extension createConnectorExtensionWithPropertyDependencies() { + final Extension extension = createConnectorExtension(); + + final ConnectorPropertyDependency dependency = new ConnectorPropertyDependency(); + dependency.setPropertyName("authType"); + dependency.setDependentValues(List.of("basic", "oauth")); + + final ConnectorProperty property = new ConnectorProperty(); + property.setName(PROPERTY_NAME); + property.setDescription(PROPERTY_DESCRIPTION); + property.setDependencies(List.of(dependency)); + + final org.apache.nifi.extension.manifest.ConnectorPropertyGroup group = new org.apache.nifi.extension.manifest.ConnectorPropertyGroup(); + group.setName(GROUP_NAME); + group.setProperties(List.of(property)); + + final org.apache.nifi.extension.manifest.ConfigurationStep step = new org.apache.nifi.extension.manifest.ConfigurationStep(); + step.setName(STEP_NAME); + step.setPropertyGroups(List.of(group)); + extension.setConfigurationSteps(List.of(step)); + + return extension; + } +} + diff --git a/nifi-mock/src/main/java/org/apache/nifi/util/MockValidationContext.java b/nifi-mock/src/main/java/org/apache/nifi/util/MockValidationContext.java index 826cea5dc3cc..6253c9e08de9 100644 --- a/nifi-mock/src/main/java/org/apache/nifi/util/MockValidationContext.java +++ b/nifi-mock/src/main/java/org/apache/nifi/util/MockValidationContext.java @@ -31,6 +31,7 @@ import org.apache.nifi.parameter.ParameterLookup; import org.apache.nifi.parameter.ParameterParser; import org.apache.nifi.parameter.ParameterReference; +import org.apache.nifi.parameter.ParameterTokenList; import java.util.Collection; import java.util.Collections; @@ -219,6 +220,13 @@ public boolean isParameterSet(final String parameterName) { return contextParameters.containsKey(parameterName) && contextParameters.get(parameterName) != null; } + @Override + public String evaluateParameters(final String value) { + final ParameterParser parameterParser = new ExpressionLanguageAgnosticParameterParser(); + final ParameterTokenList parameterTokenList = parameterParser.parseTokens(value); + return parameterTokenList.substitute(parameterLookup); + } + private String getEffectiveValue(final PropertyDescriptor descriptor) { final String configuredValue = context.getProperties().get(descriptor); return getEffectiveValue(descriptor, configuredValue); diff --git a/nifi-server-api/src/main/java/org/apache/nifi/nar/ExtensionMapping.java b/nifi-server-api/src/main/java/org/apache/nifi/nar/ExtensionMapping.java index 851fefecb744..1d4b9cd8759a 100644 --- a/nifi-server-api/src/main/java/org/apache/nifi/nar/ExtensionMapping.java +++ b/nifi-server-api/src/main/java/org/apache/nifi/nar/ExtensionMapping.java @@ -34,6 +34,7 @@ public class ExtensionMapping { private final Map> flowAnalysisRuleNames = new HashMap<>(); private final Map> parameterProviderNames = new HashMap<>(); private final Map> flowRegistryClientNames = new HashMap<>(); + private final Map> connectorNames = new HashMap<>(); private final BiFunction, Set, Set> merger = (oldValue, newValue) -> { final Set merged = new HashSet<>(); @@ -102,6 +103,16 @@ void addAllFlowRegistryClients(final BundleCoordinate coordinate, final Collecti }); } + void addConnector(final BundleCoordinate coordinate, final String connectorName) { + connectorNames.computeIfAbsent(connectorName, name -> new HashSet<>()).add(coordinate); + } + + void addAllConnectors(final BundleCoordinate coordinate, final Collection connectorNames) { + connectorNames.forEach(name -> { + addConnector(coordinate, name); + }); + } + void merge(final ExtensionMapping other) { other.getProcessorNames().forEach((name, otherCoordinates) -> { processorNames.merge(name, otherCoordinates, merger); @@ -121,6 +132,9 @@ void merge(final ExtensionMapping other) { other.getFlowRegistryClientNames().forEach((name, otherCoordinates) -> { flowRegistryClientNames.merge(name, otherCoordinates, merger); }); + other.getConnectorNames().forEach((name, otherCoordinates) -> { + connectorNames.merge(name, otherCoordinates, merger); + }); } public Map> getProcessorNames() { @@ -147,6 +161,10 @@ public Map> getFlowRegistryClientNames() { return Collections.unmodifiableMap(flowRegistryClientNames); } + public Map> getConnectorNames() { + return Collections.unmodifiableMap(connectorNames); + } + public Map> getAllExtensionNames() { final Map> extensionNames = new HashMap<>(); extensionNames.putAll(processorNames); @@ -155,6 +173,7 @@ public Map> getAllExtensionNames() { extensionNames.putAll(flowAnalysisRuleNames); extensionNames.putAll(parameterProviderNames); extensionNames.putAll(flowRegistryClientNames); + extensionNames.putAll(connectorNames); return extensionNames; } @@ -179,6 +198,9 @@ public int size() { for (final Set coordinates : flowRegistryClientNames.values()) { size += coordinates.size(); } + for (final Set coordinates : connectorNames.values()) { + size += coordinates.size(); + } return size; } @@ -189,6 +211,7 @@ public boolean isEmpty() { && reportingTaskNames.isEmpty() && flowAnalysisRuleNames.isEmpty() && parameterProviderNames.isEmpty() - && flowRegistryClientNames.isEmpty(); + && flowRegistryClientNames.isEmpty() + && connectorNames.isEmpty(); } } diff --git a/nifi-stateless/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/flow/StatelessDataflow.java b/nifi-stateless/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/flow/StatelessDataflow.java index 452b985345ac..7cc2a3eea020 100644 --- a/nifi-stateless/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/flow/StatelessDataflow.java +++ b/nifi-stateless/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/flow/StatelessDataflow.java @@ -109,4 +109,13 @@ default void shutdown() { OptionalLong getCounter(String componentId, String counterName); Map getCounters(Pattern counterNamePattern); + + /** + * Returns the latest time at which any component in the dataflow had activity, or an empty OptionalLong if there has been no activity. + * + * @return the latest activity time in milliseconds since epoch, or empty if no activity has occurred + */ + default OptionalLong getLatestActivityTime() { + return OptionalLong.empty(); + } } diff --git a/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/AllowListClassLoader.java b/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/AllowListClassLoader.java index 26ec99b86cea..3a90031d7019 100644 --- a/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/AllowListClassLoader.java +++ b/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/AllowListClassLoader.java @@ -99,6 +99,11 @@ private boolean isClassAllowed(final String name, final Class clazz) { return true; } + // TODO: Delete this + if (name.startsWith("org.mockito") || name.startsWith("net.bytebuddy") || name.startsWith("org.testcontainers")) { + return true; + } + // If the class has a module whose name is allowed, allow it. // The module is obtained by calling Class.getModule(). However, that method is only available in Java 9. // Since this codebase must be Java 8 compatible we can't make that method call. So we use Reflection to determine diff --git a/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/StatelessBootstrap.java b/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/StatelessBootstrap.java index ff2064bc895e..a0bd035eb1c9 100644 --- a/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/StatelessBootstrap.java +++ b/nifi-stateless/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/StatelessBootstrap.java @@ -149,7 +149,7 @@ public static StatelessBootstrap bootstrap(final StatelessEngineConfiguration en * @param parent the parent class loader that the given BlockListClassLoader should delegate to for classes that it does not block * @return an AllowListClassLoader that allows only the appropriate classes to be loaded from the given parent */ - protected static AllowListClassLoader createExtensionRootClassLoader(final File narDirectory, final ClassLoader parent) throws IOException { + public static AllowListClassLoader createExtensionRootClassLoader(final File narDirectory, final ClassLoader parent) throws IOException { final File[] narDirectoryFiles = narDirectory.listFiles(); if (narDirectoryFiles == null) { throw new IOException("Could not get a listing of the NAR directory"); diff --git a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/reporting/StatelessReportingTaskNode.java b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/reporting/StatelessReportingTaskNode.java index 8ec54520b5bc..a5601ca98a66 100644 --- a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/reporting/StatelessReportingTaskNode.java +++ b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/reporting/StatelessReportingTaskNode.java @@ -19,6 +19,7 @@ import org.apache.nifi.authorization.Resource; import org.apache.nifi.authorization.resource.Authorizable; +import org.apache.nifi.components.ValidationContext; import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.validation.ValidationTrigger; import org.apache.nifi.controller.LoggableComponent; @@ -50,7 +51,7 @@ public StatelessReportingTaskNode(final LoggableComponent reporti } @Override - protected List validateConfig() { + protected List validateConfig(final ValidationContext validationContext) { return Collections.emptyList(); } diff --git a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/scheduling/StatelessProcessScheduler.java b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/scheduling/StatelessProcessScheduler.java index 2f6c38ac8faa..802bbcb1ed80 100644 --- a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/scheduling/StatelessProcessScheduler.java +++ b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/controller/scheduling/StatelessProcessScheduler.java @@ -20,6 +20,7 @@ import org.apache.nifi.annotation.lifecycle.OnScheduled; import org.apache.nifi.annotation.lifecycle.OnShutdown; import org.apache.nifi.annotation.notification.PrimaryNodeState; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.components.validation.ValidationStatus; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.Funnel; @@ -128,7 +129,7 @@ public void initialize(final StatelessProcessSchedulerInitializationContext cont } @Override - public Future startProcessor(final ProcessorNode procNode, final boolean failIfStopping) { + public CompletableFuture startProcessor(final ProcessorNode procNode, final boolean failIfStopping) { final CompletableFuture future = new CompletableFuture<>(); final SchedulingAgentCallback callback = new SchedulingAgentCallback() { @Override @@ -195,7 +196,7 @@ public void onProcessorRemoved(final ProcessorNode procNode) { } @Override - public Future startStatelessGroup(final StatelessGroupNode groupNode) { + public CompletableFuture startStatelessGroup(final StatelessGroupNode groupNode) { throw new UnsupportedOperationException(); } @@ -328,12 +329,32 @@ public CompletableFuture enableControllerService(final ControllerServiceNo return service.enable(componentLifeCycleThreadPool, ADMINISTRATIVE_YIELD_MILLIS, true); } + @Override + public CompletableFuture enableControllerService(final ControllerServiceNode service, final ConfigurationContext configurationContext) { + logger.info("Enabling {}", service); + return service.enable(componentLifeCycleThreadPool, ADMINISTRATIVE_YIELD_MILLIS, true, configurationContext); + } + @Override public CompletableFuture disableControllerService(final ControllerServiceNode service) { logger.info("Disabling {}", service); return service.disable(componentLifeCycleThreadPool); } + @Override + public CompletableFuture startConnector(final ConnectorNode connectorNode) { + throw new UnsupportedOperationException(); + } + + @Override + public CompletableFuture stopConnector(final ConnectorNode connectorNode) { + throw new UnsupportedOperationException(); + } + + @Override + public void onConnectorRemoved(final ConnectorNode connectorNode) { + } + @Override public CompletableFuture disableControllerServices(final List services) { if (services == null || services.isEmpty()) { diff --git a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/engine/StatelessFlowManager.java b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/engine/StatelessFlowManager.java index 09010dd7dff4..a637745bb61e 100644 --- a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/engine/StatelessFlowManager.java +++ b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/engine/StatelessFlowManager.java @@ -21,6 +21,7 @@ import org.apache.nifi.annotation.lifecycle.OnConfigurationRestored; import org.apache.nifi.authorization.resource.Authorizable; import org.apache.nifi.bundle.BundleCoordinate; +import org.apache.nifi.components.connector.ConnectorNode; import org.apache.nifi.components.state.StateManager; import org.apache.nifi.connectable.Connectable; import org.apache.nifi.connectable.ConnectableType; @@ -225,7 +226,7 @@ public Port createLocalOutputPort(final String id, final String name) { } @Override - public ProcessGroup createProcessGroup(final String id) { + public ProcessGroup createProcessGroup(final String id, final String connectorId) { final ProcessGroup created = new StandardProcessGroup(id, statelessEngine.getControllerServiceProvider(), statelessEngine.getProcessScheduler(), statelessEngine.getPropertyEncryptor(), @@ -236,7 +237,8 @@ public ProcessGroup createProcessGroup(final String id) { new StatelessNodeTypeProvider(), null, group -> null, - statelessEngine.getAssetManager()); + statelessEngine.getAssetManager(), + connectorId); onProcessGroupAdded(created); return created; @@ -442,6 +444,21 @@ public ControllerServiceNode getRootControllerService(final String serviceIdenti public void removeRootControllerService(final ControllerServiceNode service) { } + @Override + public ConnectorNode createConnector(final String type, final String id, final BundleCoordinate coordinate, final boolean firstTimeAdded, final boolean registerLogObserver) { + throw new UnsupportedOperationException(); + } + + @Override + public List getAllConnectors() { + return List.of(); + } + + @Override + public ConnectorNode getConnector(final String id) { + return null; + } + @Override protected Authorizable getParameterContextParent() { return null; diff --git a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/flow/StandardStatelessFlow.java b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/flow/StandardStatelessFlow.java index f50913288871..5c317d7bf752 100644 --- a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/flow/StandardStatelessFlow.java +++ b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/flow/StandardStatelessFlow.java @@ -962,6 +962,11 @@ public Map getCounters(final Pattern counterNamePattern) { .collect(Collectors.toMap(Counter::getName, Counter::getValue)); } + @Override + public OptionalLong getLatestActivityTime() { + return rootGroup.getFlowFileActivity().getLatestActivityTime(); + } + private String findInstanceId(final String componentId) { return rootGroup.findAllProcessors().stream() .filter(processor -> Objects.equals(processor.getIdentifier(), componentId) || Objects.equals(processor.getVersionedComponentId().orElse(""), componentId)) diff --git a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/queue/StatelessFlowFileQueue.java b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/queue/StatelessFlowFileQueue.java index caaa607b104b..2c30d9168392 100644 --- a/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/queue/StatelessFlowFileQueue.java +++ b/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/queue/StatelessFlowFileQueue.java @@ -17,6 +17,7 @@ package org.apache.nifi.stateless.queue; +import org.apache.nifi.components.connector.DropFlowFileSummary; import org.apache.nifi.controller.queue.DropFlowFileStatus; import org.apache.nifi.controller.queue.ListFlowFileStatus; import org.apache.nifi.controller.queue.LoadBalanceCompression; @@ -32,6 +33,7 @@ import org.apache.nifi.processor.FlowFileFilter; import org.apache.nifi.util.FormatUtils; +import java.io.IOException; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; @@ -45,6 +47,7 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Predicate; public class StatelessFlowFileQueue implements DrainableFlowFileQueue { private final String identifier; @@ -317,6 +320,11 @@ public DropFlowFileStatus cancelDropFlowFileRequest(final String requestIdentifi throw new UnsupportedOperationException("Cannot drop FlowFiles from a queue in Stateless NiFi"); } + @Override + public DropFlowFileSummary dropFlowFiles(final Predicate predicate) throws IOException { + throw new UnsupportedOperationException("Cannot drop FlowFiles from a queue in Stateless NiFi"); + } + @Override public ListFlowFileStatus listFlowFiles(final String requestIdentifier, final int maxResults) { throw new UnsupportedOperationException("Cannot list FlowFiles in a queue in Stateless NiFi"); diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml index f877db14d043..fd23562cd9e9 100644 --- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/pom.xml @@ -43,6 +43,11 @@ com.fasterxml.jackson.core jackson-databind + + org.apache.nifi + nifi-connector-utils + 2.9.0-SNAPSHOT + org.apache.nifi nifi-system-test-extensions-services-api diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/AssetConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/AssetConnector.java new file mode 100644 index 000000000000..2e927bb038a7 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/AssetConnector.java @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; + +import java.util.List; +import java.util.Map; + +/** + * Example Connector used by system tests for exercising Connector Asset APIs. + *

+ * The connector defines a single configuration step with one property whose type is ASSET. + * The connector itself does not manipulate the flow; it is intended only to drive API-level tests. + */ +public class AssetConnector extends AbstractConnector { + + static final ConnectorPropertyDescriptor ASSET_PROPERTY = new ConnectorPropertyDescriptor.Builder() + .name("Test Asset") + .description("Asset used for validating Connector Asset APIs in system tests") + .type(PropertyType.ASSET) + .required(true) + .build(); + + private static final ConnectorPropertyGroup ASSET_GROUP = new ConnectorPropertyGroup.Builder() + .name("Asset Configuration") + .description("Configuration properties related to Assets") + .addProperty(ASSET_PROPERTY) + .build(); + + private static final ConfigurationStep ASSET_CONFIGURATION_STEP = new ConfigurationStep.Builder() + .name("Asset Configuration") + .description("Configuration Step that exposes an ASSET-type property") + .propertyGroups(List.of(ASSET_GROUP)) + .build(); + + @Override + public VersionedExternalFlow getInitialFlow() { + // This Connector is intended only for exercising the REST API and does not manage a flow. + return null; + } + + @Override + public void prepareForUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + // No-op: this connector does not manipulate the flow. + } + + @Override + public List getConfigurationSteps() { + return List.of(ASSET_CONFIGURATION_STEP); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + // No-op: this connector does not manipulate the flow. + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + // No-op: there is no additional behavior when the step is configured. + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map overrides, final FlowContext flowContext) { + // No additional verification is required for this simple test connector. + return List.of(); + } +} + + diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/BundleResolutionConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/BundleResolutionConnector.java new file mode 100644 index 000000000000..f15d668ef51f --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/BundleResolutionConnector.java @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.BundleCompatibility; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; + +/** + * A test connector that exercises bundle resolution capabilities. + * It creates a flow with processors using different bundle specifications to test + * how the framework handles unavailable bundles based on the configured BundleCompatability strategy. + */ +public class BundleResolutionConnector extends AbstractConnector { + + private static final String BUNDLE_COMPATABILITY_STEP = "Bundle Resolution"; + + private static final ConnectorPropertyDescriptor BUNDLE_COMPATABILITY_PROPERTY = new ConnectorPropertyDescriptor.Builder() + .name("Bundle Compatability") + .description("Specifies how bundle resolution should be handled when the specified bundle is not available.") + .required(true) + .type(PropertyType.STRING) + .allowableValues( + BundleCompatibility.REQUIRE_EXACT_BUNDLE.name(), + BundleCompatibility.RESOLVE_BUNDLE.name(), + BundleCompatibility.RESOLVE_NEWEST_BUNDLE.name() + ) + .defaultValue(BundleCompatibility.REQUIRE_EXACT_BUNDLE.name()) + .build(); + + private static final ConnectorPropertyGroup BUNDLE_PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Bundle Settings") + .description("Settings for bundle resolution behavior.") + .properties(List.of(BUNDLE_COMPATABILITY_PROPERTY)) + .build(); + + private static final ConfigurationStep BUNDLE_STEP = new ConfigurationStep.Builder() + .name(BUNDLE_COMPATABILITY_STEP) + .propertyGroups(List.of(BUNDLE_PROPERTY_GROUP)) + .build(); + + private final List configurationSteps = List.of(BUNDLE_STEP); + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + if (BUNDLE_COMPATABILITY_STEP.equals(stepName)) { + final String compatabilityValue = workingContext.getConfigurationContext() + .getProperty(BUNDLE_STEP, BUNDLE_COMPATABILITY_PROPERTY) + .getValue(); + final BundleCompatibility bundleCompatability = BundleCompatibility.valueOf(compatabilityValue); + + final VersionedExternalFlow flow = createFlowWithBundleScenarios(); + getInitializationContext().updateFlow(workingContext, flow, bundleCompatability); + } + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final VersionedProcessGroup group = VersionedFlowUtils.createProcessGroup(UUID.randomUUID().toString(), "Bundle Resolution Flow"); + + final VersionedParameter compatabilityParam = new VersionedParameter(); + compatabilityParam.setName("BUNDLE_COMPATABILITY"); + compatabilityParam.setValue(BundleCompatibility.REQUIRE_EXACT_BUNDLE.name()); + compatabilityParam.setSensitive(false); + compatabilityParam.setProvided(false); + compatabilityParam.setReferencedAssets(List.of()); + + final VersionedParameterContext parameterContext = new VersionedParameterContext(); + parameterContext.setName("Bundle Resolution Parameter Context"); + parameterContext.setParameters(Set.of(compatabilityParam)); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setParameterContexts(Map.of(parameterContext.getName(), parameterContext)); + flow.setFlowContents(group); + return flow; + } + + private VersionedExternalFlow createFlowWithBundleScenarios() { + final VersionedProcessGroup group = VersionedFlowUtils.createProcessGroup(UUID.randomUUID().toString(), "Bundle Resolution Flow"); + + // Add a processor with an unavailable bundle (fake version) that should be resolved based on BundleCompatability + // Uses the system test GenerateFlowFile processor which is available in the system test extensions bundle + final Bundle nonexistentBundle = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "0.0.0-NONEXISTENT"); + + final VersionedProcessor testProcessor = VersionedFlowUtils.addProcessor(group, + "org.apache.nifi.processors.tests.system.GenerateFlowFile", nonexistentBundle, + "GenerateFlowFile for Bundle Resolution Test", new Position(100, 100)); + testProcessor.setSchedulingPeriod("1 sec"); + testProcessor.setAutoTerminatedRelationships(Set.of("success")); + + final VersionedParameterContext parameterContext = new VersionedParameterContext(); + parameterContext.setName("Bundle Resolution Parameter Context"); + parameterContext.setParameters(Set.of()); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setParameterContexts(Map.of(parameterContext.getName(), parameterContext)); + flow.setFlowContents(group); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, + final Map propertyValueOverrides, + final FlowContext flowContext) { + return List.of(new ConfigVerificationResult.Builder() + .outcome(Outcome.SUCCESSFUL) + .subject(stepName) + .verificationStepName("Bundle Resolution Verification") + .explanation("Bundle resolution configuration verified successfully.") + .build()); + } + + @Override + public List getConfigurationSteps() { + return configurationSteps; + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) throws FlowUpdateException { + final String compatabilityValue = workingFlowContext.getConfigurationContext() + .getProperty(BUNDLE_STEP, BUNDLE_COMPATABILITY_PROPERTY) + .getValue(); + final BundleCompatibility bundleCompatability = BundleCompatibility.valueOf(compatabilityValue); + + final VersionedExternalFlow flow = createFlowWithBundleScenarios(); + getInitializationContext().updateFlow(activeFlowContext, flow, bundleCompatability); + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/CalculateConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/CalculateConnector.java new file mode 100644 index 000000000000..bb767219154d --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/CalculateConnector.java @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.InvocationFailedException; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.components.connector.StepConfigurationContext; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ProcessorFacade; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.processor.util.StandardValidators; + +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; +import java.util.List; +import java.util.Map; + +/** + * A test connector that invokes a ConnectorMethod on a CalculateProcessor using its own POJO types. + * This connector is used to test JSON marshalling of complex objects across ClassLoader boundaries. + */ +public class CalculateConnector extends AbstractConnector { + + /** + * A POJO representing a calculation request. This is intentionally a different class than + * the Processor's record type to test cross-ClassLoader JSON marshalling. + */ + public static class Calculation { + private int operand1; + private int operand2; + private String operation; + + public Calculation() { + } + + public Calculation(final int operand1, final int operand2, final String operation) { + this.operand1 = operand1; + this.operand2 = operand2; + this.operation = operation; + } + + public int getOperand1() { + return operand1; + } + + public void setOperand1(final int operand1) { + this.operand1 = operand1; + } + + public int getOperand2() { + return operand2; + } + + public void setOperand2(final int operand2) { + this.operand2 = operand2; + } + + public String getOperation() { + return operation; + } + + public void setOperation(final String operation) { + this.operation = operation; + } + } + + /** + * A POJO representing the result of a calculation. This is intentionally a different class than + * the Processor's record type to test cross-ClassLoader JSON marshalling. + */ + public static class CalculatedResult { + private Calculation calculation; + private int result; + + public Calculation getCalculation() { + return calculation; + } + + public void setCalculation(final Calculation calculation) { + this.calculation = calculation; + } + + public int getResult() { + return result; + } + + public void setResult(final int result) { + this.result = result; + } + } + + private static final ConnectorPropertyDescriptor OPERAND_1 = new ConnectorPropertyDescriptor.Builder() + .name("Operand 1") + .description("The first operand for the calculation") + .type(PropertyType.STRING) + .required(true) + .addValidator(StandardValidators.INTEGER_VALIDATOR) + .build(); + + private static final ConnectorPropertyDescriptor OPERAND_2 = new ConnectorPropertyDescriptor.Builder() + .name("Operand 2") + .description("The second operand for the calculation") + .type(PropertyType.STRING) + .required(true) + .addValidator(StandardValidators.INTEGER_VALIDATOR) + .build(); + + private static final ConnectorPropertyDescriptor OPERATION = new ConnectorPropertyDescriptor.Builder() + .name("Operation") + .description("The operation to perform (ADD, SUBTRACT, MULTIPLY, DIVIDE)") + .type(PropertyType.STRING) + .required(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + private static final ConnectorPropertyDescriptor OUTPUT_FILE = new ConnectorPropertyDescriptor.Builder() + .name("Output File") + .description("The file to write the calculation result to") + .type(PropertyType.STRING) + .required(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + private static final ConnectorPropertyGroup CALCULATION_GROUP = new ConnectorPropertyGroup.Builder() + .name("Calculation Configuration") + .description("Configuration properties for the calculation") + .properties(List.of(OPERAND_1, OPERAND_2, OPERATION, OUTPUT_FILE)) + .build(); + + private static final ConfigurationStep CALCULATION_STEP = new ConfigurationStep.Builder() + .name("Calculation") + .description("Configure the calculation parameters") + .propertyGroups(List.of(CALCULATION_GROUP)) + .build(); + + @Override + public VersionedExternalFlow getInitialFlow() { + final Bundle bundle = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "2.8.0-SNAPSHOT"); + final VersionedProcessGroup group = VersionedFlowUtils.createProcessGroup("calculate-flow-id", "Calculate Flow"); + + VersionedFlowUtils.addProcessor(group, "org.apache.nifi.processors.tests.system.Calculate", bundle, "Calculate Processor", new Position(0, 0)); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(group); + flow.setParameterContexts(Map.of()); + return flow; + } + + @Override + public List getConfigurationSteps() { + return List.of(CALCULATION_STEP); + } + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final StepConfigurationContext stepContext = workingContext.getConfigurationContext().scopedToStep(CALCULATION_STEP); + final int operand1 = stepContext.getProperty(OPERAND_1).asInteger(); + final int operand2 = stepContext.getProperty(OPERAND_2).asInteger(); + final String operation = stepContext.getProperty(OPERATION).getValue(); + final String outputFile = stepContext.getProperty(OUTPUT_FILE).getValue(); + + final ProcessorFacade processorFacade = workingContext.getRootGroup().getProcessors().stream() + .filter(p -> p.getDefinition().getType().endsWith("Calculate")) + .findFirst() + .orElseThrow(() -> new FlowUpdateException("CalculateProcessor not found in flow")); + + final Calculation calculation = new Calculation(operand1, operand2, operation); + final CalculatedResult result; + try { + result = processorFacade.invokeConnectorMethod("calculate", Map.of("calculation", calculation), CalculatedResult.class); + } catch (final InvocationFailedException e) { + throw new FlowUpdateException("Failed to invoke calculate method", e); + } + + final File file = new File(outputFile); + try (final FileWriter writer = new FileWriter(file)) { + writer.write(String.valueOf(result.getCalculation().getOperand1())); + writer.write("\n"); + writer.write(String.valueOf(result.getCalculation().getOperand2())); + writer.write("\n"); + writer.write(result.getCalculation().getOperation()); + writer.write("\n"); + writer.write(String.valueOf(result.getResult())); + } catch (final IOException e) { + throw new FlowUpdateException("Failed to write result to file", e); + } + + getLogger().info("Calculation result: {} {} {} = {}", operand1, operation, operand2, result.getResult()); + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } +} + diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/ComponentLifecycleConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/ComponentLifecycleConnector.java new file mode 100644 index 000000000000..81c6af171b13 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/ComponentLifecycleConnector.java @@ -0,0 +1,177 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.ControllerServiceAPI; +import org.apache.nifi.flow.ExecutionEngine; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; + +/** + * Test Connector designed to verify the complete component lifecycle. + * Creates a flow with: + * - A processor at the root level + * - A child process group with input and output ports + * - A processor within the child group + * - A stateless group with a processor + * + * This allows testing that start/stop operations properly handle all component types recursively. + */ +public class ComponentLifecycleConnector extends AbstractConnector { + + private static final Bundle SYSTEM_TEST_EXTENSIONS_BUNDLE = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "2.8.0-SNAPSHOT"); + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final VersionedProcessGroup rootGroup = createRootGroup(); + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + return flow; + } + + private VersionedProcessGroup createRootGroup() { + final VersionedProcessGroup rootGroup = VersionedFlowUtils.createProcessGroup(UUID.randomUUID().toString(), "Component Lifecycle Root"); + rootGroup.setPosition(new Position(0, 0)); + rootGroup.setRemoteProcessGroups(new HashSet<>()); + rootGroup.setScheduledState(ScheduledState.ENABLED); + rootGroup.setExecutionEngine(ExecutionEngine.STANDARD); + + final VersionedControllerService rootControllerService = VersionedFlowUtils.addControllerService(rootGroup, + "org.apache.nifi.cs.tests.system.StandardCountService", SYSTEM_TEST_EXTENSIONS_BUNDLE, "Root Count Service"); + rootControllerService.setScheduledState(ScheduledState.ENABLED); + final ControllerServiceAPI rootServiceApi = new ControllerServiceAPI(); + rootServiceApi.setType("org.apache.nifi.cs.tests.system.CountService"); + rootServiceApi.setBundle(SYSTEM_TEST_EXTENSIONS_BUNDLE); + rootControllerService.setControllerServiceApis(Collections.singletonList(rootServiceApi)); + + final VersionedProcessor rootProcessor = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.GenerateFlowFile", SYSTEM_TEST_EXTENSIONS_BUNDLE, "Root GenerateFlowFile", new Position(100, 100)); + rootProcessor.setSchedulingPeriod("10 sec"); + + final VersionedProcessor rootTerminateProcessor = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.TerminateFlowFile", SYSTEM_TEST_EXTENSIONS_BUNDLE, "Root TerminateFlowFile", new Position(300, 100)); + + final VersionedProcessGroup childGroup = createChildGroup(rootGroup.getIdentifier()); + rootGroup.getProcessGroups().add(childGroup); + + final VersionedPort childInputPort = childGroup.getInputPorts().iterator().next(); + final VersionedPort childOutputPort = childGroup.getOutputPorts().iterator().next(); + + VersionedFlowUtils.addConnection(rootGroup, VersionedFlowUtils.createConnectableComponent(rootProcessor), + VersionedFlowUtils.createConnectableComponent(childInputPort), Set.of("success")); + VersionedFlowUtils.addConnection(rootGroup, VersionedFlowUtils.createConnectableComponent(childOutputPort), + VersionedFlowUtils.createConnectableComponent(rootTerminateProcessor), Set.of("")); + + return rootGroup; + } + + private VersionedProcessGroup createChildGroup(final String parentGroupId) { + final VersionedProcessGroup childGroup = VersionedFlowUtils.createProcessGroup("child-group-id", "Child Group"); + childGroup.setPosition(new Position(100, 300)); + childGroup.setRemoteProcessGroups(new HashSet<>()); + childGroup.setScheduledState(ScheduledState.ENABLED); + childGroup.setExecutionEngine(ExecutionEngine.STANDARD); + childGroup.setGroupIdentifier(parentGroupId); + + final VersionedControllerService childControllerService = VersionedFlowUtils.addControllerService(childGroup, + "org.apache.nifi.cs.tests.system.StandardCountService", SYSTEM_TEST_EXTENSIONS_BUNDLE, "Child Count Service"); + childControllerService.setScheduledState(ScheduledState.ENABLED); + final ControllerServiceAPI childServiceApi = new ControllerServiceAPI(); + childServiceApi.setType("org.apache.nifi.cs.tests.system.CountService"); + childServiceApi.setBundle(SYSTEM_TEST_EXTENSIONS_BUNDLE); + childControllerService.setControllerServiceApis(Collections.singletonList(childServiceApi)); + + final VersionedPort inputPort = VersionedFlowUtils.addInputPort(childGroup, "Child Input", new Position(0, 0)); + final VersionedPort outputPort = VersionedFlowUtils.addOutputPort(childGroup, "Child Output", new Position(200, 0)); + + final VersionedProcessor childProcessor = VersionedFlowUtils.addProcessor(childGroup, + "org.apache.nifi.processors.tests.system.PassThrough", SYSTEM_TEST_EXTENSIONS_BUNDLE, "Child Terminate", new Position(100, 100)); + + final VersionedProcessGroup statelessGroup = createStatelessGroup(childGroup.getIdentifier()); + childGroup.getProcessGroups().add(statelessGroup); + + final VersionedPort statelessInputPort = statelessGroup.getInputPorts().iterator().next(); + + VersionedFlowUtils.addConnection(childGroup, VersionedFlowUtils.createConnectableComponent(inputPort), + VersionedFlowUtils.createConnectableComponent(childProcessor), Set.of("")); + VersionedFlowUtils.addConnection(childGroup, VersionedFlowUtils.createConnectableComponent(inputPort), + VersionedFlowUtils.createConnectableComponent(statelessInputPort), Set.of("")); + VersionedFlowUtils.addConnection(childGroup, VersionedFlowUtils.createConnectableComponent(childProcessor), + VersionedFlowUtils.createConnectableComponent(outputPort), Set.of("success")); + + return childGroup; + } + + private VersionedProcessGroup createStatelessGroup(final String parentGroupId) { + final VersionedProcessGroup statelessGroup = VersionedFlowUtils.createProcessGroup("stateless-group-id", "Stateless Group"); + statelessGroup.setPosition(new Position(400, 100)); + statelessGroup.setRemoteProcessGroups(new HashSet<>()); + statelessGroup.setScheduledState(ScheduledState.ENABLED); + statelessGroup.setExecutionEngine(ExecutionEngine.STATELESS); + statelessGroup.setStatelessFlowTimeout("1 min"); + statelessGroup.setGroupIdentifier(parentGroupId); + + final VersionedPort statelessInput = VersionedFlowUtils.addInputPort(statelessGroup, "Stateless Input", new Position(0, 0)); + + final VersionedProcessor statelessProcessor = VersionedFlowUtils.addProcessor(statelessGroup, + "org.apache.nifi.processors.tests.system.TerminateFlowFile", SYSTEM_TEST_EXTENSIONS_BUNDLE, "Stateless Terminate", new Position(100, 100)); + + VersionedFlowUtils.addConnection(statelessGroup, VersionedFlowUtils.createConnectableComponent(statelessInput), + VersionedFlowUtils.createConnectableComponent(statelessProcessor), Set.of("")); + + return statelessGroup; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) throws FlowUpdateException { + getInitializationContext().updateFlow(activeFlowContext, getInitialFlow()); + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/DataQueuingConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/DataQueuingConnector.java new file mode 100644 index 000000000000..1c50941c8371 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/DataQueuingConnector.java @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class DataQueuingConnector extends AbstractConnector { + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final Bundle bundle = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "2.8.0-SNAPSHOT"); + final VersionedProcessGroup rootGroup = VersionedFlowUtils.createProcessGroup("1234", "Data Queuing Connector"); + + final VersionedProcessor generate = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.GenerateFlowFile", bundle, "GenerateFlowFile", new Position(0, 0)); + generate.getProperties().put("File Size", "1 KB"); + generate.setSchedulingPeriod("100 millis"); + + final VersionedProcessor terminate = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.TerminateFlowFile", bundle, "TerminateFlowFile", new Position(0, 0)); + terminate.setScheduledState(ScheduledState.DISABLED); + + VersionedFlowUtils.addConnection(rootGroup, VersionedFlowUtils.createConnectableComponent(generate), + VersionedFlowUtils.createConnectableComponent(terminate), Set.of("success")); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + flow.setParameterContexts(Collections.emptyMap()); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) { + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/GatedDataQueuingConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/GatedDataQueuingConnector.java new file mode 100644 index 000000000000..8c7f680d86ba --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/GatedDataQueuingConnector.java @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.BundleCompatibility; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * A connector that generates FlowFiles and allows controlling when the TerminateFlowFile processor + * will process them via a gate file. If the gate file does not exist, FlowFiles will queue up. + * When the gate file exists, the TerminateFlowFile processor will process the queued FlowFiles. + *

+ * The connector uses a configuration step to specify the gate file path. The test must configure + * the connector with a gate file path and apply the update before starting the connector. + */ +public class GatedDataQueuingConnector extends AbstractConnector { + + static final ConnectorPropertyDescriptor GATE_FILE_PATH = new ConnectorPropertyDescriptor.Builder() + .name("Gate File Path") + .description("The path to the gate file. When this file exists, the TerminateFlowFile processor " + + "will process FlowFiles. When it does not exist, FlowFiles will queue up.") + .required(true) + .type(PropertyType.STRING) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + private static final ConnectorPropertyGroup PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Gate Configuration") + .addProperty(GATE_FILE_PATH) + .build(); + + private static final ConfigurationStep CONFIG_STEP = new ConfigurationStep.Builder() + .name("Gate Configuration") + .propertyGroups(List.of(PROPERTY_GROUP)) + .build(); + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final Bundle bundle = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "2.8.0"); + final VersionedProcessGroup rootGroup = VersionedFlowUtils.createProcessGroup("1234", "Gated Data Queuing Connector"); + + final VersionedProcessor generate = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.GenerateFlowFile", bundle, "GenerateFlowFile", new Position(0, 0)); + generate.getProperties().put("File Size", "1 KB"); + generate.setSchedulingPeriod("100 millis"); + + final VersionedProcessor terminate = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.TerminateFlowFile", bundle, "TerminateFlowFile", new Position(0, 0)); + + VersionedFlowUtils.addConnection(rootGroup, VersionedFlowUtils.createConnectableComponent(generate), + VersionedFlowUtils.createConnectableComponent(terminate), Set.of("success")); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + flow.setParameterContexts(Collections.emptyMap()); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(CONFIG_STEP); + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) throws FlowUpdateException { + final String gateFilePath = workingFlowContext.getConfigurationContext().getProperty(CONFIG_STEP, GATE_FILE_PATH).getValue(); + if (gateFilePath == null) { + return; + } + + final VersionedExternalFlow flow = getInitialFlow(); + final VersionedProcessGroup rootGroup = flow.getFlowContents(); + + VersionedFlowUtils.findProcessor(rootGroup, p -> p.getType().endsWith("TerminateFlowFile")) + .ifPresent(processor -> processor.getProperties().put("Gate File", gateFilePath)); + + getInitializationContext().updateFlow(activeFlowContext, flow, BundleCompatibility.RESOLVE_BUNDLE); + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/NestedProcessGroupConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/NestedProcessGroupConnector.java new file mode 100644 index 000000000000..e3903a38f1de --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/NestedProcessGroupConnector.java @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; + +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Test Connector that creates a flow with a nested (child) process group. + * Used to verify that child process groups within a connector's managed flow + * can be properly accessed via the connector API. + */ +public class NestedProcessGroupConnector extends AbstractConnector { + + public static final String CHILD_PROCESS_GROUP_ID = "child-process-group-id"; + + private final List configurationSteps = List.of(); + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final VersionedProcessGroup childGroup = new VersionedProcessGroup(); + childGroup.setIdentifier(CHILD_PROCESS_GROUP_ID); + childGroup.setName("Child Process Group"); + + final VersionedProcessGroup rootGroup = new VersionedProcessGroup(); + rootGroup.setName("Nested Flow"); + rootGroup.setProcessGroups(Set.of(childGroup)); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return configurationSteps; + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) { + } +} + diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/NopConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/NopConnector.java new file mode 100644 index 000000000000..3d28240ca256 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/NopConnector.java @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class NopConnector extends AbstractConnector { + + private static final ConnectorPropertyDescriptor IGNORED_PROPERTY = new ConnectorPropertyDescriptor.Builder() + .name("Ignored Property") + .description("This property is ignored by the NopConnector.") + .required(false) + .type(PropertyType.STRING) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + private static final ConnectorPropertyGroup IGNORED_PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Ignored Property Group") + .description("This property group is ignored by the NopConnector.") + .properties(List.of(IGNORED_PROPERTY)) + .build(); + + private static final ConnectorPropertyDescriptor SECRET_PROPERTY = new ConnectorPropertyDescriptor.Builder() + .name("Secret Property") + .description("This is a secret property that is ignored by the NopConnector.") + .required(false) + .type(PropertyType.SECRET) + .addValidator(new SuperSecretValidator()) + .build(); + private static final ConnectorPropertyGroup SECRET_PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Secret Property Group") + .description("This property group is ignored by the NopConnector.") + .properties(List.of(SECRET_PROPERTY)) + .build(); + + private static final ConfigurationStep IGNORED_STEP = new ConfigurationStep.Builder() + .name("Ignored Step") + .propertyGroups(List.of(IGNORED_PROPERTY_GROUP, SECRET_PROPERTY_GROUP)) + .build(); + + private final List configurationSteps = List.of(IGNORED_STEP); + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final VersionedProcessGroup group = new VersionedProcessGroup(); + group.setName("Nop Flow"); + + final VersionedParameter someText = new VersionedParameter(); + someText.setName("SOME_TEXT"); + someText.setValue("Lorem ipsum"); + someText.setSensitive(false); + someText.setProvided(false); + someText.setReferencedAssets(List.of()); + + final VersionedParameterContext parameterContext = new VersionedParameterContext(); + parameterContext.setName("Nop Parameter Context"); + parameterContext.setParameters(Set.of(someText)); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setParameterContexts(Map.of(parameterContext.getName(), parameterContext)); + flow.setFlowContents(group); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(new ConfigVerificationResult.Builder() + .outcome(Outcome.SUCCESSFUL) + .subject(stepName) + .verificationStepName("Nop Verification") + .explanation("Successful verification with properties: " + propertyValueOverrides) + .build()); + } + + @Override + public List getConfigurationSteps() { + return configurationSteps; + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) { + } + + private static class SuperSecretValidator implements Validator { + @Override + public ValidationResult validate(final String subject, final String input, final ValidationContext context) { + return new ValidationResult.Builder() + .subject(subject) + .input(input) + .explanation("Value must be 'supersecret'") + .valid(input.equals("supersecret")) + .build(); + } + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/ParameterContextConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/ParameterContextConnector.java new file mode 100644 index 000000000000..9d160a11f1c2 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/ParameterContextConnector.java @@ -0,0 +1,286 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.BundleCompatibility; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.ConnectorPropertyDescriptor; +import org.apache.nifi.components.connector.ConnectorPropertyGroup; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.PropertyType; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.flow.VersionedParameterContext; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.processor.util.StandardValidators; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * A test connector that creates a flow with nested Process Groups and Parameter Context inheritance. + * Used to verify that sensitive parameters and asset-referencing parameters work correctly + * when inherited from child Parameter Contexts. + * + * The flow structure: + * - Root Group (bound to Parent Parameter Context) + * - GenerateFlowFile (generates 1 FlowFile) + * - Process Group A (bound to Parent Parameter Context) + * - Input Port + * - UpdateContent (Sensitive Content = #{sensitive_param}) + * - WriteToFile (target/sensitive.txt) + * - Process Group B (bound to Parent Parameter Context) + * - Input Port + * - ReplaceWithFile (Filename = #{asset_param}) + * - WriteToFile (target/asset.txt) + * + * Parameter Context hierarchy: + * - Parent Parameter Context (inherits from Child Context A and Child Context B) + * - Child Context A: sensitive_param (sensitive) + * - Child Context B: asset_param (asset reference) + */ +public class ParameterContextConnector extends AbstractConnector { + + private static final String CONFIGURATION_STEP_NAME = "Parameter Context Configuration"; + + private static final String ROOT_GROUP_ID = "root-group"; + private static final String GROUP_A_ID = "process-group-a"; + private static final String GROUP_B_ID = "process-group-b"; + + private static final String PARENT_CONTEXT_NAME = "Parent Parameter Context"; + private static final String CHILD_CONTEXT_A_NAME = "Child Context A"; + private static final String CHILD_CONTEXT_B_NAME = "Child Context B"; + + private static final String SENSITIVE_PARAM_NAME = "sensitive_param"; + private static final String ASSET_PARAM_NAME = "asset_param"; + + static final ConnectorPropertyDescriptor SENSITIVE_VALUE = new ConnectorPropertyDescriptor.Builder() + .name("Sensitive Value") + .description("The sensitive value to be stored in the sensitive parameter") + .required(true) + .type(PropertyType.SECRET) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + static final ConnectorPropertyDescriptor ASSET_FILE = new ConnectorPropertyDescriptor.Builder() + .name("Asset File") + .description("The asset file whose contents will be used via the asset parameter") + .required(true) + .type(PropertyType.ASSET) + .build(); + + static final ConnectorPropertyDescriptor SENSITIVE_OUTPUT_FILE = new ConnectorPropertyDescriptor.Builder() + .name("Sensitive Output File") + .description("The file path where the sensitive value output will be written") + .required(true) + .type(PropertyType.STRING) + .defaultValue("target/sensitive.txt") + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + static final ConnectorPropertyDescriptor ASSET_OUTPUT_FILE = new ConnectorPropertyDescriptor.Builder() + .name("Asset Output File") + .description("The file path where the asset contents output will be written") + .required(true) + .type(PropertyType.STRING) + .defaultValue("target/asset.txt") + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .build(); + + private static final ConnectorPropertyGroup PROPERTY_GROUP = new ConnectorPropertyGroup.Builder() + .name("Parameter Context Configuration") + .description("Configuration properties for parameter context testing") + .properties(List.of(SENSITIVE_VALUE, ASSET_FILE, SENSITIVE_OUTPUT_FILE, ASSET_OUTPUT_FILE)) + .build(); + + private static final ConfigurationStep CONFIG_STEP = new ConfigurationStep.Builder() + .name(CONFIGURATION_STEP_NAME) + .description("Configure the sensitive value and asset file for parameter context testing") + .propertyGroups(List.of(PROPERTY_GROUP)) + .build(); + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) throws FlowUpdateException { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + return createEmptyFlow(); + } + + private VersionedExternalFlow createEmptyFlow() { + final VersionedProcessGroup rootGroup = VersionedFlowUtils.createProcessGroup(ROOT_GROUP_ID, "Parameter Context Test Flow"); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + flow.setParameterContexts(Collections.emptyMap()); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(CONFIG_STEP); + } + + @Override + public void applyUpdate(final FlowContext workingContext, final FlowContext activeContext) throws FlowUpdateException { + final String sensitiveValue = workingContext.getConfigurationContext().getProperty(CONFIG_STEP, SENSITIVE_VALUE).getValue(); + final String assetFilePath = workingContext.getConfigurationContext().getProperty(CONFIG_STEP, ASSET_FILE).getValue(); + final String sensitiveOutputFile = workingContext.getConfigurationContext().getProperty(CONFIG_STEP, SENSITIVE_OUTPUT_FILE).getValue(); + final String assetOutputFile = workingContext.getConfigurationContext().getProperty(CONFIG_STEP, ASSET_OUTPUT_FILE).getValue(); + + if (sensitiveValue == null || assetFilePath == null) { + return; + } + + final VersionedExternalFlow flow = createFlow(sensitiveValue, assetFilePath, sensitiveOutputFile, assetOutputFile); + getInitializationContext().updateFlow(activeContext, flow, BundleCompatibility.RESOLVE_BUNDLE); + } + + private VersionedExternalFlow createFlow(final String sensitiveValue, final String assetFilePath, + final String sensitiveOutputFile, final String assetOutputFile) { + final Bundle bundle = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "2.8.0-SNAPSHOT"); + final Map parameterContexts = createParameterContexts(sensitiveValue, assetFilePath); + + final VersionedProcessGroup rootGroup = VersionedFlowUtils.createProcessGroup(ROOT_GROUP_ID, "Parameter Context Test Flow"); + rootGroup.setParameterContextName(PARENT_CONTEXT_NAME); + + final VersionedProcessor generateProcessor = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.GenerateFlowFile", bundle, "GenerateFlowFile", new Position(0, 0)); + generateProcessor.getProperties().putAll(Map.of("Max FlowFiles", "1", "File Size", "0 B")); + generateProcessor.setSchedulingPeriod("60 sec"); + + final VersionedProcessGroup groupA = createProcessGroupA(bundle, sensitiveOutputFile); + rootGroup.getProcessGroups().add(groupA); + + final VersionedProcessGroup groupB = createProcessGroupB(bundle, assetOutputFile); + rootGroup.getProcessGroups().add(groupB); + + final VersionedPort inputPortA = groupA.getInputPorts().iterator().next(); + final VersionedPort inputPortB = groupB.getInputPorts().iterator().next(); + + VersionedFlowUtils.addConnection(rootGroup, VersionedFlowUtils.createConnectableComponent(generateProcessor), + VersionedFlowUtils.createConnectableComponent(inputPortA), Set.of("success")); + VersionedFlowUtils.addConnection(rootGroup, VersionedFlowUtils.createConnectableComponent(generateProcessor), + VersionedFlowUtils.createConnectableComponent(inputPortB), Set.of("success")); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + flow.setParameterContexts(parameterContexts); + return flow; + } + + private Map createParameterContexts(final String sensitiveValue, final String assetFilePath) { + final VersionedParameter sensitiveParam = new VersionedParameter(); + sensitiveParam.setName(SENSITIVE_PARAM_NAME); + sensitiveParam.setSensitive(true); + sensitiveParam.setValue(sensitiveValue); + sensitiveParam.setProvided(false); + sensitiveParam.setReferencedAssets(List.of()); + + final VersionedParameterContext childContextA = new VersionedParameterContext(); + childContextA.setName(CHILD_CONTEXT_A_NAME); + childContextA.setParameters(Set.of(sensitiveParam)); + + final VersionedParameter assetParam = new VersionedParameter(); + assetParam.setName(ASSET_PARAM_NAME); + assetParam.setSensitive(false); + assetParam.setValue(assetFilePath); + assetParam.setProvided(false); + assetParam.setReferencedAssets(List.of()); + + final VersionedParameterContext childContextB = new VersionedParameterContext(); + childContextB.setName(CHILD_CONTEXT_B_NAME); + childContextB.setParameters(Set.of(assetParam)); + + final VersionedParameterContext parentContext = new VersionedParameterContext(); + parentContext.setName(PARENT_CONTEXT_NAME); + parentContext.setParameters(Set.of()); + parentContext.setInheritedParameterContexts(List.of(CHILD_CONTEXT_A_NAME, CHILD_CONTEXT_B_NAME)); + + final Map contexts = new HashMap<>(); + contexts.put(CHILD_CONTEXT_A_NAME, childContextA); + contexts.put(CHILD_CONTEXT_B_NAME, childContextB); + contexts.put(PARENT_CONTEXT_NAME, parentContext); + return contexts; + } + + private VersionedProcessGroup createProcessGroupA(final Bundle bundle, final String outputFile) { + final VersionedProcessGroup groupA = VersionedFlowUtils.createProcessGroup(GROUP_A_ID, "Process Group A - Sensitive Value"); + groupA.setGroupIdentifier(ROOT_GROUP_ID); + groupA.setParameterContextName(PARENT_CONTEXT_NAME); + + final VersionedPort inputPortA = VersionedFlowUtils.addInputPort(groupA, "Input Port A", new Position(0, 0)); + + final VersionedProcessor updateContent = VersionedFlowUtils.addProcessor(groupA, + "org.apache.nifi.processors.tests.system.UpdateContent", bundle, "UpdateContent", new Position(0, 0)); + updateContent.getProperties().putAll(Map.of("Sensitive Content", "#{" + SENSITIVE_PARAM_NAME + "}", "Update Strategy", "Replace")); + + final VersionedProcessor writeToFile = VersionedFlowUtils.addProcessor(groupA, + "org.apache.nifi.processors.tests.system.WriteToFile", bundle, "WriteToFile", new Position(0, 0)); + writeToFile.getProperties().put("Filename", outputFile); + writeToFile.setAutoTerminatedRelationships(Set.of("success", "failure")); + + VersionedFlowUtils.addConnection(groupA, VersionedFlowUtils.createConnectableComponent(inputPortA), + VersionedFlowUtils.createConnectableComponent(updateContent), Set.of()); + VersionedFlowUtils.addConnection(groupA, VersionedFlowUtils.createConnectableComponent(updateContent), + VersionedFlowUtils.createConnectableComponent(writeToFile), Set.of("success")); + + return groupA; + } + + private VersionedProcessGroup createProcessGroupB(final Bundle bundle, final String outputFile) { + final VersionedProcessGroup groupB = VersionedFlowUtils.createProcessGroup(GROUP_B_ID, "Process Group B - Asset Value"); + groupB.setGroupIdentifier(ROOT_GROUP_ID); + groupB.setParameterContextName(PARENT_CONTEXT_NAME); + + final VersionedPort inputPortB = VersionedFlowUtils.addInputPort(groupB, "Input Port B", new Position(0, 0)); + + final VersionedProcessor replaceWithFile = VersionedFlowUtils.addProcessor(groupB, + "org.apache.nifi.processors.tests.system.ReplaceWithFile", bundle, "ReplaceWithFile", new Position(0, 0)); + replaceWithFile.getProperties().put("Filename", "#{" + ASSET_PARAM_NAME + "}"); + + final VersionedProcessor writeToFile = VersionedFlowUtils.addProcessor(groupB, + "org.apache.nifi.processors.tests.system.WriteToFile", bundle, "WriteToFile", new Position(0, 0)); + writeToFile.getProperties().put("Filename", outputFile); + writeToFile.setAutoTerminatedRelationships(Set.of("success", "failure")); + + VersionedFlowUtils.addConnection(groupB, VersionedFlowUtils.createConnectableComponent(inputPortB), + VersionedFlowUtils.createConnectableComponent(replaceWithFile), Set.of()); + VersionedFlowUtils.addConnection(groupB, VersionedFlowUtils.createConnectableComponent(replaceWithFile), + VersionedFlowUtils.createConnectableComponent(writeToFile), Set.of("success")); + + return groupB; + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/SelectiveDropConnector.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/SelectiveDropConnector.java new file mode 100644 index 000000000000..b1b852b18afa --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/connectors/tests/system/SelectiveDropConnector.java @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.connectors.tests.system; + +import org.apache.nifi.components.ConfigVerificationResult; +import org.apache.nifi.components.connector.AbstractConnector; +import org.apache.nifi.components.connector.ConfigurationStep; +import org.apache.nifi.components.connector.DropFlowFileSummary; +import org.apache.nifi.components.connector.FlowUpdateException; +import org.apache.nifi.components.connector.components.ConnectionFacade; +import org.apache.nifi.components.connector.components.FlowContext; +import org.apache.nifi.components.connector.components.ProcessGroupFacade; +import org.apache.nifi.components.connector.util.VersionedFlowUtils; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.Position; +import org.apache.nifi.flow.ScheduledState; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedExternalFlow; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.flowfile.FlowFile; + +import java.io.IOException; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * A Connector for testing selective dropping of FlowFiles. + * When started, it generates FlowFiles with a 'flowFileIndex' attribute. + * When stopped, it drops all FlowFiles where the flowFileIndex attribute has an even value. + */ +public class SelectiveDropConnector extends AbstractConnector { + + @Override + protected void onStepConfigured(final String stepName, final FlowContext workingContext) { + } + + @Override + public VersionedExternalFlow getInitialFlow() { + final Bundle bundle = new Bundle("org.apache.nifi", "nifi-system-test-extensions-nar", "2.7.0-SNAPSHOT"); + final VersionedProcessGroup rootGroup = VersionedFlowUtils.createProcessGroup("1234", "Selective Drop Connector"); + + final VersionedProcessor generate = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.GenerateFlowFile", bundle, "GenerateFlowFile", new Position(0, 0)); + generate.getProperties().putAll(Map.of( + "File Size", "1 B", + "Batch Size", "20000", + "Max FlowFiles", "20000", + "flowFileIndex", "${nextInt()}" + )); + generate.setSchedulingPeriod("10 sec"); + + final VersionedProcessor terminate = VersionedFlowUtils.addProcessor(rootGroup, + "org.apache.nifi.processors.tests.system.TerminateFlowFile", bundle, "TerminateFlowFile", new Position(0, 0)); + terminate.setScheduledState(ScheduledState.DISABLED); + + final VersionedConnection connection = VersionedFlowUtils.addConnection(rootGroup, + VersionedFlowUtils.createConnectableComponent(generate), VersionedFlowUtils.createConnectableComponent(terminate), Set.of("success")); + connection.setBackPressureDataSizeThreshold("100 GB"); + connection.setBackPressureObjectThreshold(100_000L); + + final VersionedExternalFlow flow = new VersionedExternalFlow(); + flow.setFlowContents(rootGroup); + flow.setParameterContexts(Collections.emptyMap()); + return flow; + } + + @Override + public List verifyConfigurationStep(final String stepName, final Map propertyValueOverrides, final FlowContext flowContext) { + return List.of(); + } + + @Override + public List getConfigurationSteps() { + return List.of(); + } + + @Override + public void applyUpdate(final FlowContext workingFlowContext, final FlowContext activeFlowContext) throws FlowUpdateException { + final VersionedExternalFlow flow = getInitialFlow(); + getInitializationContext().updateFlow(activeFlowContext, flow); + } + + @Override + public void stop(final FlowContext context) throws FlowUpdateException { + super.stop(context); + + final ProcessGroupFacade rootGroup = context.getRootGroup(); + final ConnectionFacade connection = findFirstConnection(rootGroup); + + if (connection == null) { + getLogger().warn("Could not find connection to perform selective drop"); + return; + } + + try { + final AtomicInteger dropCount = new AtomicInteger(); + final AtomicInteger seenCount = new AtomicInteger(); + final DropFlowFileSummary summary = connection.dropFlowFiles(ff -> { + final boolean shouldDrop = hasEvenFlowFileIndex(ff); + seenCount.incrementAndGet(); + if (shouldDrop) { + dropCount.incrementAndGet(); + } + return shouldDrop; + }); + getLogger().info("Selectively dropped {} out of {} FlowFiles ({} bytes) with even flowFileIndex values", + dropCount.get(), seenCount.get(), summary.getDroppedBytes()); + } catch (final IOException e) { + throw new FlowUpdateException("Failed to selectively drop FlowFiles", e); + } + } + + private boolean hasEvenFlowFileIndex(final FlowFile flowFile) { + return Integer.parseInt(flowFile.getAttribute("flowFileIndex")) % 2 == 0; + } + + private ConnectionFacade findFirstConnection(final ProcessGroupFacade group) { + for (final ConnectionFacade connection : group.getConnections()) { + return connection; + } + + for (final ProcessGroupFacade childGroup : group.getProcessGroups()) { + final ConnectionFacade found = findFirstConnection(childGroup); + if (found != null) { + return found; + } + } + + return null; + } +} diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/Calculate.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/Calculate.java new file mode 100644 index 000000000000..d8bc65b8a55f --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/Calculate.java @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.tests.system; + +import org.apache.nifi.components.connector.components.ComponentState; +import org.apache.nifi.components.connector.components.ConnectorMethod; +import org.apache.nifi.components.connector.components.MethodArgument; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.exception.ProcessException; + +/** + * A test processor that exposes a ConnectorMethod for performing calculations. + * This processor is used to test JSON marshalling of complex objects across ClassLoader boundaries. + */ +public class Calculate extends AbstractProcessor { + + @ConnectorMethod( + name = "calculate", + description = "Performs a calculation based on the provided Calculation object", + allowedStates = ComponentState.STOPPED, + arguments = { + @MethodArgument(name = "calculation", type = Calculation.class, description = "The calculation to perform", required = true) + } + ) + public Object calculate(final Calculation calculation) { + final int result = switch (calculation.operation()) { + case "ADD" -> calculation.operand1() + calculation.operand2(); + case "SUBTRACT" -> calculation.operand1() - calculation.operand2(); + case "MULTIPLY" -> calculation.operand1() * calculation.operand2(); + case "DIVIDE" -> calculation.operand1() / calculation.operand2(); + default -> throw new IllegalArgumentException("Unknown operation: " + calculation.operation()); + }; + + return new CalculatedResult(calculation, result); + } + + @Override + public void onTrigger(final ProcessContext processContext, final ProcessSession processSession) throws ProcessException { + } + + /** + * A record representing a calculation request with two operands and an operation. + */ + public record Calculation(int operand1, int operand2, String operation) { + } + + /** + * A record representing the result of a calculation, including the original calculation and the result. + */ + public record CalculatedResult(Calculation calculation, int result) { + } + +} + diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java index 734766fa37eb..9ab6fd71b18c 100644 --- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/TerminateFlowFile.java @@ -16,15 +16,43 @@ */ package org.apache.nifi.processors.tests.system; +import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.processor.AbstractProcessor; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.ProcessSession; import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +import java.io.File; +import java.util.List; public class TerminateFlowFile extends AbstractProcessor { + + public static final PropertyDescriptor GATE_FILE = new PropertyDescriptor.Builder() + .name("Gate File") + .description("An optional file path. If specified, the processor will only process FlowFiles when this file exists. " + + "If the file does not exist, the processor will yield and return without processing any data.") + .required(false) + .addValidator(StandardValidators.NON_BLANK_VALIDATOR) + .build(); + + @Override + protected List getSupportedPropertyDescriptors() { + return List.of(GATE_FILE); + } + @Override public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { + final String gateFilePath = context.getProperty(GATE_FILE).getValue(); + if (gateFilePath != null) { + final File gateFile = new File(gateFilePath); + if (!gateFile.exists()) { + context.yield(); + return; + } + } + FlowFile flowFile = session.get(); if (flowFile == null) { return; diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/UpdateContent.java b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/UpdateContent.java index 0f79a426668e..652d779fd2df 100644 --- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/UpdateContent.java +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/processors/tests/system/UpdateContent.java @@ -19,6 +19,8 @@ import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.components.PropertyDescriptor.Builder; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; import org.apache.nifi.components.Validator; import org.apache.nifi.expression.ExpressionLanguageScope; import org.apache.nifi.flowfile.FlowFile; @@ -29,7 +31,9 @@ import org.apache.nifi.processor.exception.ProcessException; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.Collections; import java.util.List; import java.util.Set; @@ -40,9 +44,17 @@ public class UpdateContent extends AbstractProcessor { .name("Content") .displayName("Content") .description("Content to set") - .required(true) + .required(false) + .addValidator(Validator.VALID) + .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) + .build(); + static final PropertyDescriptor SENSITIVE_CONTENT = new Builder() + .name("Sensitive Content") + .displayName("Sensitive Content") + .description("Sensitive content to set (use for sensitive parameter references)") + .required(false) + .sensitive(true) .addValidator(Validator.VALID) - .defaultValue("Default Content") .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); static final PropertyDescriptor UPDATE_STRATEGY = new Builder() @@ -60,7 +72,23 @@ public class UpdateContent extends AbstractProcessor { @Override protected List getSupportedPropertyDescriptors() { - return Arrays.asList(CONTENT, UPDATE_STRATEGY); + return Arrays.asList(CONTENT, SENSITIVE_CONTENT, UPDATE_STRATEGY); + } + + @Override + protected Collection customValidate(final ValidationContext context) { + final List results = new ArrayList<>(); + final boolean hasContent = context.getProperty(CONTENT).isSet(); + final boolean hasSensitiveContent = context.getProperty(SENSITIVE_CONTENT).isSet(); + + if (hasContent == hasSensitiveContent) { + results.add(new ValidationResult.Builder() + .subject("Content") + .valid(false) + .explanation("Exactly one of 'Content' or 'Sensitive Content' must be set") + .build()); + } + return results; } @Override @@ -75,7 +103,12 @@ public void onTrigger(final ProcessContext context, final ProcessSession session return; } - final String content = context.getProperty(CONTENT).evaluateAttributeExpressions(flowFile).getValue(); + final String content; + if (context.getProperty(CONTENT).isSet()) { + content = context.getProperty(CONTENT).evaluateAttributeExpressions(flowFile).getValue(); + } else { + content = context.getProperty(SENSITIVE_CONTENT).evaluateAttributeExpressions(flowFile).getValue(); + } final byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); final String strategy = context.getProperty(UPDATE_STRATEGY).getValue(); diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector new file mode 100644 index 000000000000..a2d46427433a --- /dev/null +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.components.connector.Connector @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +org.apache.nifi.connectors.tests.system.AssetConnector +org.apache.nifi.connectors.tests.system.BundleResolutionConnector +org.apache.nifi.connectors.tests.system.CalculateConnector +org.apache.nifi.connectors.tests.system.ComponentLifecycleConnector +org.apache.nifi.connectors.tests.system.DataQueuingConnector +org.apache.nifi.connectors.tests.system.GatedDataQueuingConnector +org.apache.nifi.connectors.tests.system.NestedProcessGroupConnector +org.apache.nifi.connectors.tests.system.NopConnector +org.apache.nifi.connectors.tests.system.ParameterContextConnector +org.apache.nifi.connectors.tests.system.SelectiveDropConnector diff --git a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor index a12f954cb84a..dab323f15001 100644 --- a/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor +++ b/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor @@ -63,3 +63,4 @@ org.apache.nifi.processors.tests.system.WriteFlowFileCountToFile org.apache.nifi.processors.tests.system.WriteLifecycleEvents org.apache.nifi.processors.tests.system.WriteToFile org.apache.nifi.processors.tests.system.YieldSource +org.apache.nifi.processors.tests.system.Calculate \ No newline at end of file diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiClientUtil.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiClientUtil.java index 2c3181a96db6..053615d2121a 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiClientUtil.java +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiClientUtil.java @@ -17,6 +17,7 @@ package org.apache.nifi.tests.system; import org.apache.nifi.cluster.coordination.node.NodeConnectionState; +import org.apache.nifi.components.connector.ConnectorState; import org.apache.nifi.controller.AbstractPort; import org.apache.nifi.controller.queue.LoadBalanceCompression; import org.apache.nifi.controller.queue.LoadBalanceStrategy; @@ -27,6 +28,7 @@ import org.apache.nifi.scheduling.ExecutionNode; import org.apache.nifi.stream.io.StreamUtils; import org.apache.nifi.toolkit.client.ConnectionClient; +import org.apache.nifi.toolkit.client.ConnectorClient; import org.apache.nifi.toolkit.client.NiFiClient; import org.apache.nifi.toolkit.client.NiFiClientException; import org.apache.nifi.toolkit.client.ProcessorClient; @@ -34,8 +36,11 @@ import org.apache.nifi.web.api.dto.AssetReferenceDTO; import org.apache.nifi.web.api.dto.BundleDTO; import org.apache.nifi.web.api.dto.ConfigVerificationResultDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; import org.apache.nifi.web.api.dto.ConnectableDTO; import org.apache.nifi.web.api.dto.ConnectionDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; import org.apache.nifi.web.api.dto.ControllerServiceDTO; import org.apache.nifi.web.api.dto.CounterDTO; import org.apache.nifi.web.api.dto.CountersSnapshotDTO; @@ -54,10 +59,12 @@ import org.apache.nifi.web.api.dto.ProcessGroupDTO; import org.apache.nifi.web.api.dto.ProcessorConfigDTO; import org.apache.nifi.web.api.dto.ProcessorDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; import org.apache.nifi.web.api.dto.RemoteProcessGroupDTO; import org.apache.nifi.web.api.dto.ReportingTaskDTO; import org.apache.nifi.web.api.dto.RevisionDTO; import org.apache.nifi.web.api.dto.VerifyConfigRequestDTO; +import org.apache.nifi.web.api.dto.VerifyConnectorConfigStepRequestDTO; import org.apache.nifi.web.api.dto.VersionControlInformationDTO; import org.apache.nifi.web.api.dto.VersionedFlowDTO; import org.apache.nifi.web.api.dto.flow.FlowDTO; @@ -69,8 +76,11 @@ import org.apache.nifi.web.api.dto.status.ProcessGroupStatusSnapshotDTO; import org.apache.nifi.web.api.dto.status.ProcessorStatusSnapshotDTO; import org.apache.nifi.web.api.entity.ActivateControllerServicesEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; import org.apache.nifi.web.api.entity.ConnectionEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorsEntity; import org.apache.nifi.web.api.entity.ControllerServiceEntity; import org.apache.nifi.web.api.entity.ControllerServiceRunStatusEntity; import org.apache.nifi.web.api.entity.ControllerServicesEntity; @@ -113,6 +123,7 @@ import org.apache.nifi.web.api.entity.ScheduleComponentsEntity; import org.apache.nifi.web.api.entity.StartVersionControlRequestEntity; import org.apache.nifi.web.api.entity.VerifyConfigRequestEntity; +import org.apache.nifi.web.api.entity.VerifyConnectorConfigStepRequestEntity; import org.apache.nifi.web.api.entity.VersionControlInformationEntity; import org.apache.nifi.web.api.entity.VersionedFlowUpdateRequestEntity; import org.junit.jupiter.api.Assertions; @@ -163,6 +174,12 @@ private ConnectionClient getConnectionClient() { return client; } + private ConnectorClient getConnectorClient() { + final ConnectorClient client = nifiClient.getConnectorClient(); + client.acknowledgeDisconnectedNode(); + return client; + } + public ProcessorEntity startProcessor(final ProcessorEntity currentEntity) throws NiFiClientException, IOException, InterruptedException { waitForValidationCompleted(currentEntity); @@ -225,6 +242,225 @@ private String simpleName(final String type) { return type.substring(lastIndex + 1); } + public ConnectorEntity createConnector(final String simpleTypeName) throws NiFiClientException, IOException { + return createConnector(NiFiSystemIT.TEST_CONNECTORS_PACKAGE + "." + simpleTypeName, NiFiSystemIT.NIFI_GROUP_ID, NiFiSystemIT.TEST_EXTENSIONS_ARTIFACT_ID, nifiVersion); + } + + public ConnectorEntity createConnector(final String type, final String bundleGroupId, final String artifactId, final String version) throws NiFiClientException, IOException { + final ConnectorDTO dto = new ConnectorDTO(); + dto.setType(type); + + final BundleDTO bundle = new BundleDTO(); + bundle.setGroup(bundleGroupId); + bundle.setArtifact(artifactId); + bundle.setVersion(version); + dto.setBundle(bundle); + + final ConnectorEntity entity = new ConnectorEntity(); + entity.setComponent(dto); + entity.setRevision(createNewRevision()); + entity.setDisconnectedNodeAcknowledged(true); + + final ConnectorEntity connector = getConnectorClient().createConnector(entity); + logger.info("Created Connector [type={}, id={}, name={}] for Test [{}]", simpleName(type), connector.getId(), connector.getComponent().getName(), testName); + return connector; + } + + public ConfigurationStepEntity configureConnector(final String connectorId, final String configurationStepName, + final Map properties) throws NiFiClientException, IOException { + final ConnectorEntity connectorEntity = getConnectorClient().getConnector(connectorId); + return configureConnector(connectorEntity, configurationStepName, properties); + } + + public ConfigurationStepEntity configureConnector(final ConnectorEntity connectorEntity, final String configurationStepName, + final Map properties) throws NiFiClientException, IOException { + final Map propertyValues = properties.entrySet().stream() + .collect(Collectors.toMap(Map.Entry::getKey, entry -> createStringLiteralValueReference(entry.getValue()))); + + final PropertyGroupConfigurationDTO propertyGroupConfig = new PropertyGroupConfigurationDTO(); + propertyGroupConfig.setPropertyValues(propertyValues); + + final ConfigurationStepConfigurationDTO stepConfig = new ConfigurationStepConfigurationDTO(); + stepConfig.setConfigurationStepName(configurationStepName); + stepConfig.setPropertyGroupConfigurations(Collections.singletonList(propertyGroupConfig)); + + final ConfigurationStepEntity stepEntity = new ConfigurationStepEntity(); + stepEntity.setParentConnectorId(connectorEntity.getId()); + stepEntity.setParentConnectorRevision(connectorEntity.getRevision()); + stepEntity.setConfigurationStep(stepConfig); + stepEntity.setDisconnectedNodeAcknowledged(true); + + final ConfigurationStepEntity result = getConnectorClient().updateConfigurationStep(stepEntity); + logger.info("Configured Connector [id={}, step={}] for Test [{}]", connectorEntity.getId(), configurationStepName, testName); + return result; + } + + private ConnectorValueReferenceDTO createStringLiteralValueReference(final String value) { + final ConnectorValueReferenceDTO valueRef = new ConnectorValueReferenceDTO(); + valueRef.setValueType("STRING_LITERAL"); + valueRef.setValue(value); + return valueRef; + } + + public ConnectorValueReferenceDTO createSecretValueReference(final String secretProviderId, final String secretName, final String fullyQualifiedSecretName) { + final ConnectorValueReferenceDTO valueRef = new ConnectorValueReferenceDTO(); + valueRef.setValueType("SECRET_REFERENCE"); + valueRef.setSecretProviderId(secretProviderId); + valueRef.setSecretName(secretName); + valueRef.setFullyQualifiedSecretName(fullyQualifiedSecretName); + return valueRef; + } + + public ConfigurationStepEntity configureConnectorWithReferences(final String connectorId, final String configurationStepName, + final Map propertyValues) throws NiFiClientException, IOException { + final ConnectorEntity connectorEntity = getConnectorClient().getConnector(connectorId); + return configureConnectorWithReferences(connectorEntity, configurationStepName, propertyValues); + } + + public ConfigurationStepEntity configureConnectorWithReferences(final ConnectorEntity connectorEntity, final String configurationStepName, + final Map propertyValues) throws NiFiClientException, IOException { + final PropertyGroupConfigurationDTO propertyGroupConfig = new PropertyGroupConfigurationDTO(); + propertyGroupConfig.setPropertyValues(propertyValues); + + final ConfigurationStepConfigurationDTO stepConfig = new ConfigurationStepConfigurationDTO(); + stepConfig.setConfigurationStepName(configurationStepName); + stepConfig.setPropertyGroupConfigurations(Collections.singletonList(propertyGroupConfig)); + + final ConfigurationStepEntity stepEntity = new ConfigurationStepEntity(); + stepEntity.setParentConnectorId(connectorEntity.getId()); + stepEntity.setParentConnectorRevision(connectorEntity.getRevision()); + stepEntity.setConfigurationStep(stepConfig); + stepEntity.setDisconnectedNodeAcknowledged(true); + + final ConfigurationStepEntity result = getConnectorClient().updateConfigurationStep(stepEntity); + logger.info("Configured Connector [id={}, step={}] for Test [{}]", connectorEntity.getId(), configurationStepName, testName); + return result; + } + + public void applyConnectorUpdate(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException, InterruptedException { + final ConnectorClient client = nifiClient.getConnectorClient(); + final ConnectorEntity initialEntity = client.getConnector(connectorEntity.getId()); + final ConnectorState initialState = ConnectorState.valueOf(initialEntity.getComponent().getState()); + client.applyUpdate(connectorEntity); + + while (true) { + final ConnectorEntity currentEntity = client.getConnector(connectorEntity.getId()); + final String state = currentEntity.getComponent().getState(); + final ConnectorState currentState = ConnectorState.valueOf(state); + + switch (currentState) { + case UPDATE_FAILED: + throw new IllegalStateException("Connector failed to update"); + case UPDATED: + case UPDATING: + case PREPARING_FOR_UPDATE: + logger.debug("Waiting for Connector [id={}] to finish updating (current state={})...", connectorEntity.getId(), state); + break; + default: + if (initialState == currentState) { + logger.info("Connector [id={}] has successfully updated and now has state of {}", connectorEntity.getId(), state); + return; + } + logger.debug("Waiting for Connector [id={}] to return to initial state of {} (current state={})...", connectorEntity.getId(), initialState, state); + break; + } + + // Wait 250 milliseconds before polling again + Thread.sleep(250L); + } + } + + public void waitForValidConnector(final String connectorId) throws NiFiClientException, IOException, InterruptedException { + waitForConnectorValidationStatus(connectorId, "VALID"); + } + + public void waitForInvalidConnector(final String connectorId) throws NiFiClientException, IOException, InterruptedException { + waitForConnectorValidationStatus(connectorId, "INVALID"); + } + + public void waitForConnectorValidationStatus(final String connectorId, final String expectedStatus) throws NiFiClientException, IOException, InterruptedException { + int iteration = 0; + while (true) { + final ConnectorEntity entity = getConnectorClient().getConnector(connectorId); + final String validationStatus = entity.getComponent().getValidationStatus(); + if (expectedStatus.equals(validationStatus)) { + return; + } + + if ("VALIDATING".equals(validationStatus)) { + logger.debug("Waiting for Connector {} to finish validating...", connectorId); + } else if (iteration++ % 30 == 0) { // Every 3 seconds log status + logger.info("Connector with ID {} has validation status {} but expected {}. Validation errors: {}", + connectorId, validationStatus, expectedStatus, entity.getComponent().getValidationErrors()); + } + + Thread.sleep(100L); + } + } + + public void stopConnectors() throws NiFiClientException, IOException, InterruptedException { + final ConnectorsEntity connectorsEntity = nifiClient.getFlowClient().getConnectors(); + for (final ConnectorEntity connector : connectorsEntity.getConnectors()) { + getConnectorClient().stopConnector(connector); + waitForConnectorStopped(connector.getId()); + } + } + + public void startConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException, InterruptedException { + startConnector(connectorEntity.getId()); + } + + public void startConnector(final String connectorId) throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity entity = getConnectorClient().getConnector(connectorId); + getConnectorClient().startConnector(entity); + waitForConnectorState(connectorId, ConnectorState.RUNNING); + } + + public void stopConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException, InterruptedException { + stopConnector(connectorEntity.getId()); + } + + public void stopConnector(final String connectorId) throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity entity = getConnectorClient().getConnector(connectorId); + getConnectorClient().stopConnector(entity); + waitForConnectorStopped(connectorId); + } + + public void waitForConnectorStopped(final String connectorId) throws NiFiClientException, IOException, InterruptedException { + waitForConnectorState(connectorId, ConnectorState.STOPPED); + } + + public void waitForConnectorState(final String connectorId, final ConnectorState desiredState) throws InterruptedException, NiFiClientException, IOException { + int iteration = 0; + while (true) { + final ConnectorEntity entity = getConnectorClient().getConnector(connectorId); + final String state = entity.getComponent().getState(); + if (desiredState.name().equals(state)) { + return; + } + + if (iteration++ % 30 == 0) { // Every 3 seconds log status + logger.info("Connector with ID {} has state {} but waiting for state {}.", connectorId, state, desiredState); + } + + Thread.sleep(100L); + } + } + + public ConnectorEntity drainConnector(final String connectorId) throws NiFiClientException, IOException { + final ConnectorEntity entity = getConnectorClient().getConnector(connectorId); + return getConnectorClient().drainConnector(entity); + } + + public ConnectorEntity cancelDrain(final String connectorId) throws NiFiClientException, IOException { + final ConnectorEntity entity = getConnectorClient().getConnector(connectorId); + return getConnectorClient().cancelDrain(entity); + } + + public void waitForConnectorDraining(final String connectorId) throws NiFiClientException, IOException, InterruptedException { + waitForConnectorState(connectorId, ConnectorState.DRAINING); + } + public ParameterProviderEntity createParameterProvider(final String simpleTypeName) throws NiFiClientException, IOException { return createParameterProvider(NiFiSystemIT.TEST_PARAM_PROVIDERS_PACKAGE + "." + simpleTypeName, NiFiSystemIT.NIFI_GROUP_ID, NiFiSystemIT.TEST_EXTENSIONS_ARTIFACT_ID, nifiVersion); } @@ -1232,6 +1468,42 @@ public void deleteControllerLevelServices() throws NiFiClientException, IOExcept } } + public void deleteConnectors() throws NiFiClientException, IOException { + final ConnectorsEntity connectors = nifiClient.getFlowClient().getConnectors(); + for (final ConnectorEntity connector : connectors.getConnectors()) { + purgeConnectorFlowFiles(connector.getId()); + connector.setDisconnectedNodeAcknowledged(true); + nifiClient.getConnectorClient().deleteConnector(connector); + } + } + + public DropRequestEntity purgeConnectorFlowFiles(final String connectorId) throws NiFiClientException, IOException { + final ConnectorClient connectorClient = getConnectorClient(); + final long maxTimestamp = System.currentTimeMillis() + TimeUnit.MINUTES.toMillis(1L); + + DropRequestEntity requestEntity = connectorClient.createPurgeRequest(connectorId); + try { + while (requestEntity.getDropRequest().getPercentCompleted() < 100) { + if (System.currentTimeMillis() > maxTimestamp) { + throw new IOException("Timed out waiting for Connector " + connectorId + " to purge FlowFiles"); + } + + try { + Thread.sleep(50L); + } catch (final InterruptedException ie) { + Thread.currentThread().interrupt(); + return null; + } + + requestEntity = connectorClient.getPurgeRequest(connectorId, requestEntity.getDropRequest().getId()); + } + } finally { + requestEntity = connectorClient.deletePurgeRequest(connectorId, requestEntity.getDropRequest().getId()); + } + + return requestEntity; + } + public void waitForControllerServiceRunStatus(final String id, final String requestedRunStatus) throws NiFiClientException, IOException { final long maxTimestamp = System.currentTimeMillis() + TimeUnit.MINUTES.toMillis(2L); logger.info("Waiting for Controller Service {} to have a Run Status of {}", id, requestedRunStatus); @@ -1930,6 +2202,49 @@ public List verifyFlowAnalysisRuleConfig(final Stri return results.getRequest().getResults(); } + public List verifyConnectorStepConfig(final String connectorId, final String configurationStepName, + final Map properties) throws NiFiClientException, IOException, InterruptedException { + + final Map propertyValues = properties.entrySet().stream() + .collect(Collectors.toMap(Map.Entry::getKey, entry -> createStringLiteralValueReference(entry.getValue()))); + + return verifyConnectorStepConfigWithReferences(connectorId, configurationStepName, propertyValues); + } + + public List verifyConnectorStepConfigWithReferences(final String connectorId, final String configurationStepName, + final Map propertyValues) throws NiFiClientException, IOException, InterruptedException { + + final PropertyGroupConfigurationDTO propertyGroupConfig = new PropertyGroupConfigurationDTO(); + propertyGroupConfig.setPropertyValues(propertyValues); + + final ConfigurationStepConfigurationDTO stepConfig = new ConfigurationStepConfigurationDTO(); + stepConfig.setConfigurationStepName(configurationStepName); + stepConfig.setPropertyGroupConfigurations(Collections.singletonList(propertyGroupConfig)); + + final VerifyConnectorConfigStepRequestDTO requestDto = new VerifyConnectorConfigStepRequestDTO(); + requestDto.setConnectorId(connectorId); + requestDto.setConfigurationStepName(configurationStepName); + requestDto.setConfigurationStep(stepConfig); + + final VerifyConnectorConfigStepRequestEntity verificationRequest = new VerifyConnectorConfigStepRequestEntity(); + verificationRequest.setRequest(requestDto); + + VerifyConnectorConfigStepRequestEntity results = getConnectorClient().submitConfigStepVerificationRequest(verificationRequest); + while (!results.getRequest().isComplete()) { + Thread.sleep(50L); + results = getConnectorClient().getConfigStepVerificationRequest(connectorId, configurationStepName, results.getRequest().getRequestId()); + } + + final String failureReason = results.getRequest().getFailureReason(); + if (failureReason != null) { + throw new IllegalStateException("Configuration step verification failed: " + failureReason); + } + + getConnectorClient().deleteConfigStepVerificationRequest(connectorId, configurationStepName, results.getRequest().getRequestId()); + + return results.getRequest().getResults(); + } + public ReportingTaskEntity createReportingTask(final String type, final String bundleGroupId, final String artifactId, final String version) throws NiFiClientException, IOException { final ReportingTaskDTO dto = new ReportingTaskDTO(); diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java index d9d5e522d700..3562174747b8 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java @@ -89,6 +89,7 @@ public abstract class NiFiSystemIT implements NiFiInstanceProvider { public static final String TEST_PYTHON_EXTENSIONS_ARTIFACT_ID = "python-extensions"; public static final String TEST_PARAM_PROVIDERS_PACKAGE = "org.apache.nifi.parameter.tests.system"; public static final String TEST_PROCESSORS_PACKAGE = "org.apache.nifi.processors.tests.system"; + public static final String TEST_CONNECTORS_PACKAGE = "org.apache.nifi.connectors.tests.system"; public static final String TEST_CS_PACKAGE = "org.apache.nifi.cs.tests.system"; public static final String TEST_REPORTING_TASK_PACKAGE = "org.apache.nifi.reporting"; public static final String TEST_FLOW_ANALYSIS_RULE_PACKAGE = "org.apache.nifi.flowanalysis"; @@ -260,12 +261,14 @@ protected void destroyFlow() throws NiFiClientException, IOException, Interrupte getClientUtil().disableControllerLevelServices(); getClientUtil().disableFlowAnalysisRules(); getClientUtil().stopTransmitting("root"); + getClientUtil().stopConnectors(); getClientUtil().deleteAll("root"); getClientUtil().deleteControllerLevelServices(); getClientUtil().deleteReportingTasks(); getClientUtil().deleteFlowAnalysisRules(); getClientUtil().deleteParameterContexts(); getClientUtil().deleteParameterProviders(); + getClientUtil().deleteConnectors(); logger.info("Finished destroyFlow"); } @@ -466,6 +469,29 @@ protected void waitForQueueCount(final String connectionId, final int queueSize) logger.info("Queue Count for Connection {} is now {}", connectionId, queueSize); } + protected void waitForConnectorMinQueueCount(final String connectorId, final int minQueueSize) throws InterruptedException { + logger.info("Waiting for Queue Count of at least {} in Connector {}", minQueueSize, connectorId); + + waitFor(() -> { + try { + final ProcessGroupStatusEntity statusEntity = getNifiClient().getConnectorClient().getStatus(connectorId, true); + final int currentSize = statusEntity.getProcessGroupStatus().getAggregateSnapshot().getFlowFilesQueued(); + logEverySecond("Current Queue Size for Connector {} = {}, Waiting for at least {}", connectorId, currentSize, minQueueSize); + return currentSize >= minQueueSize; + } catch (final Exception e) { + logger.error("Failed to get connector queue count", e); + return false; + } + }); + + logger.info("Queue Count for Connector {} is now at least {}", connectorId, minQueueSize); + } + + protected int getConnectorQueuedFlowFileCount(final String connectorId) throws NiFiClientException, IOException { + final ProcessGroupStatusEntity statusEntity = getNifiClient().getConnectorClient().getStatus(connectorId, true); + return statusEntity.getProcessGroupStatus().getAggregateSnapshot().getFlowFilesQueued(); + } + private void waitForQueueCountToMatch(final String connectionId, final Predicate test, final String queueSizeDescription) throws InterruptedException { waitFor(() -> { final ConnectionStatusEntity statusEntity = getConnectionStatus(connectionId); diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/SpawnedStandaloneNiFiInstanceFactory.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/SpawnedStandaloneNiFiInstanceFactory.java index 5474a54a46e0..588ddd63d168 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/SpawnedStandaloneNiFiInstanceFactory.java +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/SpawnedStandaloneNiFiInstanceFactory.java @@ -18,9 +18,11 @@ import org.apache.nifi.bootstrap.command.process.ManagementServerAddressProvider; import org.apache.nifi.bootstrap.command.process.ProcessBuilderProvider; +import org.apache.nifi.bootstrap.command.process.ProcessHandleManagementServerAddressProvider; import org.apache.nifi.bootstrap.command.process.StandardManagementServerAddressProvider; import org.apache.nifi.bootstrap.command.process.StandardProcessBuilderProvider; import org.apache.nifi.bootstrap.configuration.ConfigurationProvider; +import org.apache.nifi.bootstrap.configuration.ManagementServerPath; import org.apache.nifi.bootstrap.configuration.StandardConfigurationProvider; import org.apache.nifi.toolkit.client.NiFiClient; import org.apache.nifi.toolkit.client.NiFiClientConfig; @@ -38,7 +40,13 @@ import java.io.InputStream; import java.io.OutputStream; import java.io.PrintWriter; +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; +import java.time.Duration; import java.util.Collections; import java.util.HashMap; import java.util.Map; @@ -448,6 +456,8 @@ public void quarantineTroubleshootingInfo(final File destinationDir, final Throw if (process == null) { logger.warn("NiFi instance is not running so will not capture diagnostics for {}", getInstanceDirectory()); + } else { + captureDiagnostics(destinationDir); } final File causeFile = new File(destinationDir, "test-failure-stack-trace.txt"); @@ -456,6 +466,43 @@ public void quarantineTroubleshootingInfo(final File destinationDir, final Throw } } + private void captureDiagnostics(final File destinationDir) { + final ProcessHandle processHandle = process.toHandle(); + final ManagementServerAddressProvider addressProvider = new ProcessHandleManagementServerAddressProvider(processHandle); + final Optional managementServerAddress = addressProvider.getAddress(); + + if (managementServerAddress.isEmpty()) { + logger.warn("Could not determine management server address for NiFi instance {}, will not capture diagnostics", getInstanceDirectory()); + return; + } + + final String diagnosticsUri = "http://%s%s?verbose=true".formatted(managementServerAddress.get(), ManagementServerPath.HEALTH_DIAGNOSTICS.getPath()); + logger.info("Capturing diagnostics from {} for failed test", diagnosticsUri); + + try (final HttpClient httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build()) { + final HttpRequest request = HttpRequest.newBuilder() + .uri(URI.create(diagnosticsUri)) + .timeout(Duration.ofSeconds(30)) + .GET() + .build(); + + final HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)); + final int statusCode = response.statusCode(); + + if (statusCode == 200) { + final File diagnosticsFile = new File(destinationDir, "diagnostics.txt"); + try (final FileWriter writer = new FileWriter(diagnosticsFile, StandardCharsets.UTF_8)) { + writer.write(response.body()); + } + logger.info("Diagnostics captured successfully to {}", diagnosticsFile.getAbsolutePath()); + } else { + logger.warn("Failed to capture diagnostics from {}. HTTP status: {}", diagnosticsUri, statusCode); + } + } catch (final Exception e) { + logger.warn("Failed to capture diagnostics from {}", diagnosticsUri, e); + } + } + public NiFiClient createClient() throws IOException { final Properties nifiProperties = getProperties(); final String httpsPort = nifiProperties.getProperty("nifi.web.https.port"); diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/TroubleshootingTestWatcher.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/TroubleshootingTestWatcher.java index b2c1ab078f46..f800b8b5b9c3 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/TroubleshootingTestWatcher.java +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/TroubleshootingTestWatcher.java @@ -16,6 +16,7 @@ */ package org.apache.nifi.tests.system; +import org.junit.jupiter.api.extension.AfterTestExecutionCallback; import org.junit.jupiter.api.extension.ExtensionContext; import org.junit.jupiter.api.extension.TestWatcher; import org.slf4j.Logger; @@ -26,39 +27,103 @@ import java.util.Optional; /** - * Test Watcher responsible for writing troubleshooting logs + * Test Watcher responsible for writing troubleshooting logs. + *

+ * This extension implements {@link AfterTestExecutionCallback} to capture diagnostics from the running + * NiFi instance BEFORE the {@code @AfterEach} method tears it down. The JUnit 5 execution order is: + *

    + *
  1. {@code @BeforeEach}
    2. + *
  2. Test method (may fail)
    3. + *
  3. {@code AfterTestExecutionCallback.afterTestExecution()} - Diagnostics captured here while NiFi is still running
    4. + *
  4. {@code @AfterEach} - NiFi instance is torn down here
    5. + *
  5. {@code TestWatcher.testFailed()} - Too late for live diagnostics
    6. + *
*/ -public class TroubleshootingTestWatcher implements TestWatcher { +public class TroubleshootingTestWatcher implements TestWatcher, AfterTestExecutionCallback { private static final Logger logger = LoggerFactory.getLogger(TroubleshootingTestWatcher.class); + private static final ExtensionContext.Namespace NAMESPACE = ExtensionContext.Namespace.create(TroubleshootingTestWatcher.class); + private static final String QUARANTINE_DIR_KEY = "quarantineDir"; @Override - public void testFailed(final ExtensionContext context, final Throwable cause) { + public void afterTestExecution(final ExtensionContext context) { + // Check if the test failed by looking for an execution exception + final Optional executionException = context.getExecutionException(); + if (executionException.isEmpty()) { + return; // Test passed, no need to capture diagnostics + } + final Optional optionalTestInstance = context.getTestInstance(); - if (optionalTestInstance.isPresent()) { - final Object testInstance = optionalTestInstance.get(); - if (testInstance instanceof NiFiInstanceProvider) { - final NiFiInstanceProvider provider = (NiFiInstanceProvider) testInstance; - final String displayName = context.getDisplayName(); - try { - final String testClassName = context.getTestClass().map(Class::getSimpleName).orElse("TestClassUnknown"); - final File dir = quarantineTroubleshootingInfo(provider, testClassName, displayName, cause); - logger.info("Test Failed [{}]: Troubleshooting information stored [{}]", displayName, dir.getAbsolutePath()); - } catch (final Exception e) { - logger.error("Test Failed [{}]: Troubleshooting information not stored", displayName, e); + if (optionalTestInstance.isEmpty()) { + return; + } + + final Object testInstance = optionalTestInstance.get(); + if (!(testInstance instanceof NiFiInstanceProvider provider)) { + return; + } + + final NiFiInstance instance = provider.getNiFiInstance(); + if (instance == null) { + logger.warn("Test failed but NiFi Instance is not available for capturing diagnostics"); + return; + } + + final String displayName = context.getDisplayName(); + final String testClassName = context.getTestClass().map(Class::getSimpleName).orElse("TestClassUnknown"); + final Throwable cause = executionException.get(); + + try { + final File quarantineDir = quarantineTroubleshootingInfo(instance, testClassName, displayName, cause); + // Store the directory so testFailed() can log it + context.getStore(NAMESPACE).put(QUARANTINE_DIR_KEY, quarantineDir); + logger.info("Test Failed [{}]: Troubleshooting information (including live diagnostics) stored [{}]", displayName, quarantineDir.getAbsolutePath()); + } catch (final Exception e) { + logger.error("Test Failed [{}]: Failed to capture troubleshooting information", displayName, e); + } + } + + @Override + public void testFailed(final ExtensionContext context, final Throwable cause) { + // Diagnostics have already been captured in afterTestExecution() while NiFi was still running. + // This method is called after @AfterEach, so we just check if we already captured the info. + final File quarantineDir = context.getStore(NAMESPACE).get(QUARANTINE_DIR_KEY, File.class); + if (quarantineDir == null) { + // Diagnostics weren't captured in afterTestExecution - try to capture what we can now + final Optional optionalTestInstance = context.getTestInstance(); + if (optionalTestInstance.isPresent()) { + final Object testInstance = optionalTestInstance.get(); + if (testInstance instanceof NiFiInstanceProvider provider) { + final String displayName = context.getDisplayName(); + try { + final String testClassName = context.getTestClass().map(Class::getSimpleName).orElse("TestClassUnknown"); + final File dir = quarantineTroubleshootingInfoFallback(provider, testClassName, displayName, cause); + logger.info("Test Failed [{}]: Troubleshooting information stored (without live diagnostics) [{}]", displayName, dir.getAbsolutePath()); + } catch (final Exception e) { + logger.error("Test Failed [{}]: Troubleshooting information not stored", displayName, e); + } } } } } - private File quarantineTroubleshootingInfo(final NiFiInstanceProvider provider, final String testClassName, final String methodName, final Throwable failureCause) throws IOException { + private File quarantineTroubleshootingInfo(final NiFiInstance instance, final String testClassName, + final String methodName, final Throwable failureCause) throws IOException { + final File troubleshooting = new File("target/troubleshooting"); + final String quarantineDirName = testClassName + "-" + methodName.replace("()", ""); + final File quarantineDir = new File(troubleshooting, quarantineDirName); + quarantineDir.mkdirs(); + + instance.quarantineTroubleshootingInfo(quarantineDir, failureCause); + return quarantineDir; + } + + private File quarantineTroubleshootingInfoFallback(final NiFiInstanceProvider provider, final String testClassName, + final String methodName, final Throwable failureCause) throws IOException { NiFiInstance instance = provider.getNiFiInstance(); - // The teardown method may or may not have already run at this point. If it has, the instance will be null. - // In that case, just create a new instance and use it - it will map to the same directories. if (instance == null) { - logger.warn("While capturing troubleshooting info for {}, the NiFi Instance is not available. Will create a new one for Diagnostics purposes, but some of the diagnostics may be less " + - "accurate, since it's not the same instance that ran the test", methodName); - + logger.warn("While capturing troubleshooting info for {}, the NiFi Instance is not available. Will create a new one for Diagnostics purposes, but live diagnostics will not be available " + + "since it's not the same instance that ran the test", methodName); instance = provider.getInstanceFactory().createInstance(); } diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/FlowSynchronizationIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/FlowSynchronizationIT.java index acf3b1c9dadb..1365dcb3e37f 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/FlowSynchronizationIT.java +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/FlowSynchronizationIT.java @@ -688,6 +688,9 @@ public void testAddControllerServiceReferencingExistingService() throws NiFiClie @Test public void testUnnecessaryProcessorsAndConnectionsRemoved() throws NiFiClientException, IOException, InterruptedException { + // We make assertions about the number of components in the flow so make sure that we start with an empty flow. + destroyFlow(); + final ProcessorEntity generate = getClientUtil().createProcessor("GenerateFlowFile"); getClientUtil().updateProcessorSchedulingPeriod(generate, "10 mins"); diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorAssetsIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorAssetsIT.java new file mode 100644 index 000000000000..f628b5a4dd1c --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorAssetsIT.java @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.tests.system.NiFiInstanceFactory; +import org.apache.nifi.toolkit.client.ConnectorClient; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.util.file.FileUtils; +import org.apache.nifi.web.api.dto.AssetReferenceDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.AssetsEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Clustered variants of the connector system tests that verify connector assets + * are synchronized across all cluster nodes. + * + * The structure of these tests is based on the clustered parameter context + * asset synchronization tests in {@code ClusteredParameterContextIT}. + */ +public class ClusteredConnectorAssetsIT extends ConnectorAssetsIT { + + @Override + public NiFiInstanceFactory getInstanceFactory() { + return createTwoNodeInstanceFactory(); + } + + @Test + public void testSynchronizeConnectorAssets() throws NiFiClientException, IOException, InterruptedException { + waitForAllNodesConnected(); + + // Create a Connector instance using the AssetConnector test extension + final ConnectorEntity connector = getClientUtil().createConnector("AssetConnector"); + assertNotNull(connector); + assertNotNull(connector.getId()); + + final String connectorId = connector.getId(); + + // Upload an Asset to the Connector + final File assetFile = new File("src/test/resources/sample-assets/helloworld.txt"); + final ConnectorClient connectorClient = getNifiClient().getConnectorClient(); + final AssetEntity assetEntity = connectorClient.createAsset(connectorId, assetFile.getName(), assetFile); + + assertNotNull(assetEntity); + assertNotNull(assetEntity.getAsset()); + assertNotNull(assetEntity.getAsset().getId()); + assertEquals(assetFile.getName(), assetEntity.getAsset().getName()); + + final String uploadedAssetId = assetEntity.getAsset().getId(); + + // List the Connector's Assets and verify that the uploaded Asset is present + final AssetsEntity assetsEntity = connectorClient.getAssets(connectorId); + assertNotNull(assetsEntity); + assertNotNull(assetsEntity.getAssets()); + assertFalse(assetsEntity.getAssets().isEmpty()); + + final boolean assetFound = assetsEntity.getAssets().stream() + .filter(a -> a.getAsset() != null) + .anyMatch(a -> uploadedAssetId.equals(a.getAsset().getId())); + + assertTrue(assetFound); + + // Check that the asset exists in the connector_assets directory for each node + final File node1Dir = getNiFiInstance().getNodeInstance(1).getInstanceDirectory(); + final File node1AssetsDir = new File(node1Dir, "connector_assets"); + final File node1ConnectorDir = new File(node1AssetsDir, connectorId); + assertTrue(node1ConnectorDir.exists()); + + final File node2Dir = getNiFiInstance().getNodeInstance(2).getInstanceDirectory(); + final File node2AssetsDir = new File(node2Dir, "connector_assets"); + final File node2ConnectorDir = new File(node2AssetsDir, connectorId); + assertTrue(node2ConnectorDir.exists()); + + final File[] node1AssetIdDirs = node1ConnectorDir.listFiles(); + assertNotNull(node1AssetIdDirs); + assertEquals(1, node1AssetIdDirs.length); + + final File[] node2AssetIdDirs = node2ConnectorDir.listFiles(); + assertNotNull(node2AssetIdDirs); + assertEquals(1, node2AssetIdDirs.length); + + // Configure the connector's "Test Asset" property to reference the uploaded asset + final ConnectorValueReferenceDTO assetValueReference = new ConnectorValueReferenceDTO(); + assetValueReference.setValueType("ASSET_REFERENCE"); + assetValueReference.setAssetReferences(List.of(new AssetReferenceDTO(uploadedAssetId))); + getClientUtil().configureConnectorWithReferences(connectorId, "Asset Configuration", Map.of("Test Asset", assetValueReference)); + + // Apply the updates to the connector + getClientUtil().applyConnectorUpdate(connector); + + // Start the connector before disconnecting node 2 + getClientUtil().startConnector(connectorId); + + // Stop node 2 and delete its connector_assets directory + disconnectNode(2); + getNiFiInstance().getNodeInstance(2).stop(); + + FileUtils.deleteFilesInDir(node2AssetsDir, (dir, name) -> true, null, true, true); + assertTrue(node2AssetsDir.delete()); + assertFalse(node2AssetsDir.exists()); + + // Start node 2 again and wait for it to rejoin the cluster + getNiFiInstance().getNodeInstance(2).start(true); + waitForAllNodesConnected(); + + // Verify that the connector state is RUNNING after node 2 rejoins + getClientUtil().waitForConnectorState(connectorId, ConnectorState.RUNNING); + getClientUtil().waitForValidConnector(connectorId); + + // Verify node 2 connector assets directory is recreated and contains the expected asset + assertTrue(node2AssetsDir.exists()); + assertTrue(node2ConnectorDir.exists()); + + final File[] node2AssetIdDirsAfterRestart = node2ConnectorDir.listFiles(); + assertNotNull(node2AssetIdDirsAfterRestart); + assertEquals(1, node2AssetIdDirsAfterRestart.length); + assertEquals(uploadedAssetId, node2AssetIdDirsAfterRestart[0].getName()); + + // Verify that the Asset is still returned from the Connector's Assets list + final AssetsEntity assetsAfterRestart = connectorClient.getAssets(connectorId); + assertNotNull(assetsAfterRestart); + assertNotNull(assetsAfterRestart.getAssets()); + + final boolean assetStillPresent = assetsAfterRestart.getAssets().stream() + .filter(a -> a.getAsset() != null) + .anyMatch(a -> uploadedAssetId.equals(a.getAsset().getId())); + + assertTrue(assetStillPresent); + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorDrainIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorDrainIT.java new file mode 100644 index 000000000000..9ee3fbd09b2a --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorDrainIT.java @@ -0,0 +1,228 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.tests.system.NiFiInstanceFactory; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Cluster-specific tests for connector draining functionality. + * These tests verify that draining works correctly when nodes complete draining at different times. + */ +public class ClusteredConnectorDrainIT extends NiFiSystemIT { + + private static final Logger logger = LoggerFactory.getLogger(ClusteredConnectorDrainIT.class); + + @Override + public NiFiInstanceFactory getInstanceFactory() { + return createTwoNodeInstanceFactory(); + } + + /** + * Tests that when draining in a cluster: + * 1. Create gate file on Node 1 only - Node 1 finishes draining (STOPPED), Node 2 still DRAINING + * 2. Aggregate state should be DRAINING (since at least one node is still draining) + * 3. Create gate file on Node 2 - both nodes finish draining + * 4. Aggregate state should be STOPPED + */ + @Test + public void testDrainWithNodeCompletingAtDifferentTimes() throws NiFiClientException, IOException, InterruptedException { + final File node1InstanceDir = getNiFiInstance().getNodeInstance(1).getInstanceDirectory(); + final File node2InstanceDir = getNiFiInstance().getNodeInstance(2).getInstanceDirectory(); + final File node1GateFile = new File(node1InstanceDir, "gate-file.txt"); + final File node2GateFile = new File(node2InstanceDir, "gate-file.txt"); + node1GateFile.deleteOnExit(); + node2GateFile.deleteOnExit(); + deleteIfExists(node1GateFile); + deleteIfExists(node2GateFile); + + logger.info("Creating GatedDataQueuingConnector"); + final ConnectorEntity connector = getClientUtil().createConnector("GatedDataQueuingConnector"); + assertNotNull(connector); + final String connectorId = connector.getId(); + + final String gateFilePath = "./gate-file.txt"; + logger.info("Configuring connector {} with gate file path: {}", connectorId, gateFilePath); + getClientUtil().configureConnector(connector, "Gate Configuration", Map.of("Gate File Path", gateFilePath)); + getClientUtil().applyConnectorUpdate(connector); + + logger.info("Starting connector {}", connectorId); + getClientUtil().startConnector(connectorId); + + logger.info("Waiting for at least 5 FlowFiles to queue"); + waitForConnectorMinQueueCount(connectorId, 5); + + logger.info("Stopping connector {}", connectorId); + getClientUtil().stopConnector(connectorId); + getClientUtil().waitForConnectorStopped(connectorId); + + final int queuedCountBeforeDrain = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count before drain: {}", queuedCountBeforeDrain); + assertTrue(queuedCountBeforeDrain > 0); + + logger.info("Initiating drain for connector {}", connectorId); + getClientUtil().drainConnector(connectorId); + + logger.info("Waiting for aggregate connector state to be DRAINING"); + getClientUtil().waitForConnectorDraining(connectorId); + + logger.info("Creating gate file for Node 1 only: {}", node1GateFile.getAbsolutePath()); + assertTrue(node1GateFile.createNewFile()); + + logger.info("Waiting for Node 1 to finish draining and become STOPPED"); + waitForNodeConnectorState(1, connectorId, ConnectorState.STOPPED); + waitForNodeConnectorState(2, connectorId, ConnectorState.DRAINING); + + logger.info("Verifying aggregate state is still DRAINING (Node 2 still draining)"); + final ConnectorEntity aggregateConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + assertEquals(ConnectorState.DRAINING.name(), aggregateConnector.getComponent().getState()); + + logger.info("Creating gate file for Node 2: {}", node2GateFile.getAbsolutePath()); + assertTrue(node2GateFile.createNewFile()); + + logger.info("Waiting for aggregate state to become STOPPED"); + waitFor(() -> { + try { + final ConnectorEntity entity = getNifiClient().getConnectorClient().getConnector(connectorId); + return ConnectorState.STOPPED.name().equals(entity.getComponent().getState()); + } catch (final Exception e) { + return false; + } + }); + + final ConnectorEntity finalConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + logger.info("Final aggregate connector state: {}", finalConnector.getComponent().getState()); + assertEquals(ConnectorState.STOPPED.name(), finalConnector.getComponent().getState()); + + final int finalQueuedCount = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Final queued FlowFile count: {}", finalQueuedCount); + assertEquals(0, finalQueuedCount); + + logger.info("testDrainWithNodeCompletingAtDifferentTimes completed successfully"); + } + + /** + * Tests that when canceling drain in a cluster where one node has already finished: + * 1. Create gate file on Node 1 only - Node 1 finishes draining (STOPPED), Node 2 still DRAINING + * 2. Aggregate state should be DRAINING + * 3. Cancel drain - Node 2 should stop draining + * 4. Aggregate state should be STOPPED + * 5. Data should still be queued (from Node 2) + */ + @Test + public void testCancelDrainWithOneNodeAlreadyComplete() throws NiFiClientException, IOException, InterruptedException { + final File node1InstanceDir = getNiFiInstance().getNodeInstance(1).getInstanceDirectory(); + final File node2InstanceDir = getNiFiInstance().getNodeInstance(2).getInstanceDirectory(); + final File node1GateFile = new File(node1InstanceDir, "gate-file-cancel.txt"); + final File node2GateFile = new File(node2InstanceDir, "gate-file-cancel.txt"); + node1GateFile.deleteOnExit(); + node2GateFile.deleteOnExit(); + deleteIfExists(node1GateFile); + deleteIfExists(node2GateFile); + + logger.info("Creating GatedDataQueuingConnector"); + final ConnectorEntity connector = getClientUtil().createConnector("GatedDataQueuingConnector"); + assertNotNull(connector); + final String connectorId = connector.getId(); + + final String gateFilePath = "./gate-file-cancel.txt"; + logger.info("Configuring connector {} with gate file path: {}", connectorId, gateFilePath); + getClientUtil().configureConnector(connector, "Gate Configuration", Map.of("Gate File Path", gateFilePath)); + getClientUtil().applyConnectorUpdate(connector); + + logger.info("Starting connector {}", connectorId); + getClientUtil().startConnector(connectorId); + + logger.info("Waiting for at least 5 FlowFiles to queue"); + waitForConnectorMinQueueCount(connectorId, 5); + + logger.info("Stopping connector {}", connectorId); + getClientUtil().stopConnector(connectorId); + getClientUtil().waitForConnectorStopped(connectorId); + + final int queuedCountBeforeDrain = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count before drain: {}", queuedCountBeforeDrain); + assertTrue(queuedCountBeforeDrain > 0); + + logger.info("Initiating drain for connector {}", connectorId); + getClientUtil().drainConnector(connectorId); + + logger.info("Waiting for aggregate connector state to be DRAINING"); + getClientUtil().waitForConnectorDraining(connectorId); + + logger.info("Creating gate file for Node 1 only: {}", node1GateFile.getAbsolutePath()); + assertTrue(node1GateFile.createNewFile()); + + logger.info("Waiting for Node 1 to finish draining and become STOPPED"); + waitForNodeConnectorState(1, connectorId, ConnectorState.STOPPED); + + logger.info("Verifying aggregate state is still DRAINING (Node 2 still draining)"); + final ConnectorEntity aggregateBeforeCancel = getNifiClient().getConnectorClient().getConnector(connectorId); + assertEquals(ConnectorState.DRAINING.name(), aggregateBeforeCancel.getComponent().getState()); + + logger.info("Canceling drain for connector {}", connectorId); + getClientUtil().cancelDrain(connectorId); + + logger.info("Waiting for aggregate state to become STOPPED after cancel"); + getClientUtil().waitForConnectorStopped(connectorId); + + final ConnectorEntity finalConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + logger.info("Final aggregate connector state: {}", finalConnector.getComponent().getState()); + assertEquals(ConnectorState.STOPPED.name(), finalConnector.getComponent().getState()); + + final int finalQueuedCount = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Final queued FlowFile count after cancel (should still have data from Node 2): {}", finalQueuedCount); + assertTrue(finalQueuedCount > 0); + + logger.info("testCancelDrainWithOneNodeAlreadyComplete completed successfully"); + } + + private void waitForNodeConnectorState(final int nodeIndex, final String connectorId, final ConnectorState expectedState) throws InterruptedException { + logger.info("Waiting for Node {} connector {} to reach state {}", nodeIndex, connectorId, expectedState); + waitFor(() -> { + try { + switchClientToNode(nodeIndex); + final ConnectorEntity entity = getNifiClient().getConnectorClient(DO_NOT_REPLICATE).getConnector(connectorId); + return expectedState.name().equals(entity.getComponent().getState()); + } catch (final Exception e) { + return false; + } + }); + logger.info("Node {} connector {} reached state {}", nodeIndex, connectorId, expectedState); + } + + private void deleteIfExists(final File file) { + if (file.exists()) { + assertTrue(file.delete()); + } + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorIT.java new file mode 100644 index 000000000000..b07646dd179b --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ClusteredConnectorIT.java @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import jakarta.ws.rs.NotFoundException; +import org.apache.nifi.tests.system.NiFiInstanceFactory; +import org.apache.nifi.toolkit.client.ConnectorClient; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.NodeEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.fail; + +public class ClusteredConnectorIT extends ConnectorCrudIT { + private static final Logger logger = LoggerFactory.getLogger(ClusteredConnectorIT.class); + + @Override + public NiFiInstanceFactory getInstanceFactory() { + return createTwoNodeInstanceFactory(); + } + + @Test + public void testModifiedWhenDisconnected() throws NiFiClientException, IOException, InterruptedException { + // Create a new connector + final ConnectorEntity connector = getClientUtil().createConnector("NopConnector"); + assertNotNull(connector); + logger.info("Created Connector with ID {}", connector.getId()); + + // Configure the connector with an initial value and apply the update + getClientUtil().configureConnector(connector, "Ignored Step", Map.of("Ignored Property", "Initial Value")); + getClientUtil().applyConnectorUpdate(connector); + logger.info("Configured Connector with initial value and applied the update"); + + // Make requests to each node to ensure that the config has been applied + switchClientToNode(1); + ConnectorEntity node1Connector = getNifiClient().getConnectorClient(DO_NOT_REPLICATE).getConnector(connector.getId()); + assertActiveConfigurationValue(node1Connector, "Ignored Property", "Initial Value"); + assertWorkingConfigurationValue(node1Connector, "Ignored Property", "Initial Value"); + + switchClientToNode(2); + ConnectorEntity node2Connector = getNifiClient().getConnectorClient(DO_NOT_REPLICATE).getConnector(connector.getId()); + assertActiveConfigurationValue(node2Connector, "Ignored Property", "Initial Value"); + assertWorkingConfigurationValue(node2Connector, "Ignored Property", "Initial Value"); + logger.info("Validated initial configuration on both nodes"); + + // Disconnect node 2 + switchClientToNode(1); + disconnectNode(2); + logger.info("Disconnected Node 2"); + + // Make changes on node 1 and apply the change + getClientUtil().configureConnector(connector.getId(), "Ignored Step", Map.of("Ignored Property", "Applied While Disconnected")); + getNifiClient().getConnectorClient().applyUpdate(connector); + logger.info("Applied configuration change on Node 1 while Node 2 is disconnected"); + + // Configure the connector to yet another value on node 1 but do NOT apply the change + getClientUtil().configureConnector(connector.getId(), "Ignored Step", Map.of("Ignored Property", "Working Only Value")); + logger.info("Configured working configuration on Node 1 without applying the change"); + + // Verify node 1 has the expected active and working configurations + node1Connector = getNifiClient().getConnectorClient(DO_NOT_REPLICATE).getConnector(connector.getId()); + assertActiveConfigurationValue(node1Connector, "Ignored Property", "Applied While Disconnected"); + assertWorkingConfigurationValue(node1Connector, "Ignored Property", "Working Only Value"); + logger.info("Validated active and working configuration on Node 1"); + + // Reconnect Node 2 + reconnectNode(2); + waitForAllNodesConnected(); + logger.info("Reconnected Node 2"); + + // Make requests to node 2 to ensure that it properly inherited both the working and active context configuration + switchClientToNode(2); + waitFor(() -> { + try { + final ConnectorEntity latestNode2Connector = getNifiClient().getConnectorClient(DO_NOT_REPLICATE).getConnector(connector.getId()); + final String activeValue = getConfigurationValue(latestNode2Connector.getComponent().getActiveConfiguration(), "Ignored Property"); + final String workingValue = getConfigurationValue(latestNode2Connector.getComponent().getWorkingConfiguration(), "Ignored Property"); + return "Applied While Disconnected".equals(activeValue) && "Working Only Value".equals(workingValue); + } catch (final Exception e) { + return false; + } + }); + logger.info("Validated that Node 2 has received updated configuration after reconnection"); + + final ConnectorEntity finalNode2Connector = getNifiClient().getConnectorClient(DO_NOT_REPLICATE).getConnector(connector.getId()); + assertActiveConfigurationValue(finalNode2Connector, "Ignored Property", "Applied While Disconnected"); + assertWorkingConfigurationValue(finalNode2Connector, "Ignored Property", "Working Only Value"); + logger.info("Validated active and working configuration on Node 2"); + } + + @Test + public void testDeleteConnectorOnConnect() throws NiFiClientException, IOException, InterruptedException { + // Create Connector + final ConnectorEntity connector = getClientUtil().createConnector("DataQueuingConnector"); + assertNotNull(connector); + + // Disconnect node 2 + disconnectNode(2); + + // Should not be able to delete connector + final ConnectorClient connectorClient = getNifiClient().getConnectorClient(); + assertThrows(NiFiClientException.class, () -> connectorClient.deleteConnector(connector)); + + final NodeEntity node2Entity = getNodeEntity(2); + getNifiClient().getControllerClient().deleteNode(node2Entity.getNode().getNodeId()); + + // Should now be able to delete connector + connectorClient.deleteConnector(connector); + + // Should now be able to add node 2 back + getNiFiInstance().getNodeInstance(2).stop(); + getNiFiInstance().getNodeInstance(2).start(true); + waitForAllNodesConnected(); + + switchClientToNode(2); + + // We should get a 404 + try { + getNifiClient().getConnectorClient().getConnector(connector.getId()); + fail("Expected NiFiClientException but it was not thrown"); + } catch (final NiFiClientException e) { + assertInstanceOf(NotFoundException.class, e.getCause()); + } + } + + + @Test + public void testDeleteConnectorOnConnectWithDataQueued() throws NiFiClientException, IOException, InterruptedException { + // Create Connector + final ConnectorEntity connector = getClientUtil().createConnector("DataQueuingConnector"); + assertNotNull(connector); + + getNifiClient().getConnectorClient().startConnector(connector); + + Thread.sleep(1000L); // Wait 1 second to allow some data to queue + + // Disconnect node 2 + disconnectNode(2); + getNiFiInstance().getNodeInstance(2).stop(); + + // Should not be able to delete connector + final ConnectorClient connectorClient = getNifiClient().getConnectorClient(); + assertThrows(NiFiClientException.class, () -> connectorClient.deleteConnector(connector)); + + // Remove node 2 from cluster. + final NodeEntity node2Entity = getNodeEntity(2); + getNifiClient().getControllerClient().deleteNode(node2Entity.getNode().getNodeId()); + + // We cannot delete the connector directly because it has data queued. Stop Node 1, delete the flow.json.gz file, and restart Node 1. + getNiFiInstance().getNodeInstance(1).stop(); + final File node1InstanceDir = getNiFiInstance().getNodeInstance(1).getInstanceDirectory(); + final File node1ConfDir = new File(node1InstanceDir, "conf"); + final File flowJson = new File(node1ConfDir, "flow.json.gz"); + Files.delete(flowJson.toPath()); + + getNiFiInstance().getNodeInstance(1).start(true); + waitForCoordinatorElected(); + + // Should now be able to add node 2 back + getNiFiInstance().getNodeInstance(2).start(true); + waitForAllNodesConnected(); + + switchClientToNode(2); + + // We should get a 404 + try { + getNifiClient().getConnectorClient().getConnector(connector.getId()); + fail("Expected NiFiClientException but it was not thrown"); + } catch (final NiFiClientException e) { + assertInstanceOf(NotFoundException.class, e.getCause()); + } + } + + + private void assertActiveConfigurationValue(final ConnectorEntity connector, final String propertyName, final String expectedValue) { + final String actualValue = getConfigurationValue(connector.getComponent().getActiveConfiguration(), propertyName); + assertEquals(expectedValue, actualValue, "Active configuration property '" + propertyName + "' did not match expected value"); + } + + private void assertWorkingConfigurationValue(final ConnectorEntity connector, final String propertyName, final String expectedValue) { + final String actualValue = getConfigurationValue(connector.getComponent().getWorkingConfiguration(), propertyName); + assertEquals(expectedValue, actualValue, "Working configuration property '" + propertyName + "' did not match expected value"); + } + + private String getConfigurationValue(final ConnectorConfigurationDTO configuration, final String propertyName) { + final Map propertyValues = configuration.getConfigurationStepConfigurations().getFirst() + .getPropertyGroupConfigurations().getFirst().getPropertyValues(); + final ConnectorValueReferenceDTO valueRef = propertyValues.get(propertyName); + return valueRef == null ? null : valueRef.getValue(); + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorAssetsIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorAssetsIT.java new file mode 100644 index 000000000000..40a2bce775df --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorAssetsIT.java @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.ConnectorClient; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.AssetReferenceDTO; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.AssetsEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class ConnectorAssetsIT extends NiFiSystemIT { + + @Test + public void testCreateConnectorAndUploadAsset() throws NiFiClientException, IOException, InterruptedException { + // Create a Connector instance using the AssetConnector test extension + final ConnectorEntity connector = getClientUtil().createConnector("AssetConnector"); + assertNotNull(connector); + assertNotNull(connector.getId()); + + final String connectorId = connector.getId(); + + // Upload an Asset to the Connector + final File assetFile = new File("src/test/resources/sample-assets/helloworld.txt"); + final ConnectorClient connectorClient = getNifiClient().getConnectorClient(); + final AssetEntity assetEntity = connectorClient.createAsset(connectorId, assetFile.getName(), assetFile); + + assertNotNull(assetEntity); + assertNotNull(assetEntity.getAsset()); + assertNotNull(assetEntity.getAsset().getId()); + assertEquals(assetFile.getName(), assetEntity.getAsset().getName()); + + // List the Connector's Assets and verify that the uploaded Asset is present + final AssetsEntity assetsEntity = connectorClient.getAssets(connectorId); + assertNotNull(assetsEntity); + assertNotNull(assetsEntity.getAssets()); + assertFalse(assetsEntity.getAssets().isEmpty()); + + final String uploadedAssetId = assetEntity.getAsset().getId(); + final boolean assetFound = assetsEntity.getAssets().stream() + .filter(a -> a.getAsset() != null) + .anyMatch(a -> uploadedAssetId.equals(a.getAsset().getId())); + + assertTrue(assetFound); + + // Update the configuration step to reference the uploaded Asset in the Test Asset property + final String configurationStepName = "Asset Configuration"; + final String propertyGroupName = "Asset Configuration"; + final String propertyName = "Test Asset"; + + final ConfigurationStepEntity configurationStepEntity = connectorClient.getConfigurationStep(connectorId, configurationStepName); + assertNotNull(configurationStepEntity); + assertNotNull(configurationStepEntity.getConfigurationStep()); + + final ConfigurationStepConfigurationDTO configurationStepConfiguration = configurationStepEntity.getConfigurationStep(); + assertNotNull(configurationStepConfiguration.getPropertyGroupConfigurations()); + assertFalse(configurationStepConfiguration.getPropertyGroupConfigurations().isEmpty()); + + PropertyGroupConfigurationDTO assetConfigurationGroup = null; + for (final PropertyGroupConfigurationDTO group : configurationStepConfiguration.getPropertyGroupConfigurations()) { + if (propertyGroupName.equals(group.getPropertyGroupName())) { + assetConfigurationGroup = group; + break; + } + } + assertNotNull(assetConfigurationGroup); + + Map propertyValues = assetConfigurationGroup.getPropertyValues(); + if (propertyValues == null) { + propertyValues = new HashMap<>(); + assetConfigurationGroup.setPropertyValues(propertyValues); + } + + final ConnectorValueReferenceDTO assetValueReference = new ConnectorValueReferenceDTO(); + assetValueReference.setValueType("ASSET_REFERENCE"); + + final AssetReferenceDTO assetReferenceDTO = new AssetReferenceDTO(uploadedAssetId); + assetValueReference.setAssetReferences(List.of(assetReferenceDTO)); + + propertyValues.put(propertyName, assetValueReference); + + final ConfigurationStepEntity updatedConfigurationStepEntity = connectorClient.updateConfigurationStep(configurationStepEntity); + assertNotNull(updatedConfigurationStepEntity); + + // Retrieve the Connector configuration and verify that the Test Asset property has the Asset reference set + final ConnectorEntity connectorWithConfiguration = connectorClient.getConnector(connectorId); + assertNotNull(connectorWithConfiguration); + assertNotNull(connectorWithConfiguration.getComponent()); + + final ConnectorConfigurationDTO workingConfiguration = connectorWithConfiguration.getComponent().getWorkingConfiguration(); + assertNotNull(workingConfiguration); + assertNotNull(workingConfiguration.getConfigurationStepConfigurations()); + assertFalse(workingConfiguration.getConfigurationStepConfigurations().isEmpty()); + + ConfigurationStepConfigurationDTO configuredStep = null; + for (final ConfigurationStepConfigurationDTO stepConfiguration : workingConfiguration.getConfigurationStepConfigurations()) { + if (configurationStepName.equals(stepConfiguration.getConfigurationStepName())) { + configuredStep = stepConfiguration; + break; + } + } + assertNotNull(configuredStep); + assertNotNull(configuredStep.getPropertyGroupConfigurations()); + assertFalse(configuredStep.getPropertyGroupConfigurations().isEmpty()); + + PropertyGroupConfigurationDTO configuredAssetGroup = null; + for (final PropertyGroupConfigurationDTO group : configuredStep.getPropertyGroupConfigurations()) { + if (propertyGroupName.equals(group.getPropertyGroupName())) { + configuredAssetGroup = group; + break; + } + } + assertNotNull(configuredAssetGroup); + assertNotNull(configuredAssetGroup.getPropertyValues()); + + final ConnectorValueReferenceDTO configuredAssetValue = configuredAssetGroup.getPropertyValues().get(propertyName); + assertNotNull(configuredAssetValue); + assertEquals("ASSET_REFERENCE", configuredAssetValue.getValueType()); + assertNotNull(configuredAssetValue.getAssetReferences()); + assertFalse(configuredAssetValue.getAssetReferences().isEmpty()); + assertEquals(uploadedAssetId, configuredAssetValue.getAssetReferences().getFirst().getId()); + + // Update the configuration step again to remove the Asset reference + final ConfigurationStepEntity configurationStepEntityWithoutAsset = connectorClient.getConfigurationStep(connectorId, configurationStepName); + assertNotNull(configurationStepEntityWithoutAsset); + assertNotNull(configurationStepEntityWithoutAsset.getConfigurationStep()); + + final ConfigurationStepConfigurationDTO configurationStepConfigurationWithoutAsset = configurationStepEntityWithoutAsset.getConfigurationStep(); + assertNotNull(configurationStepConfigurationWithoutAsset.getPropertyGroupConfigurations()); + + PropertyGroupConfigurationDTO groupWithoutAsset = null; + for (final PropertyGroupConfigurationDTO group : configurationStepConfigurationWithoutAsset.getPropertyGroupConfigurations()) { + if (propertyGroupName.equals(group.getPropertyGroupName())) { + groupWithoutAsset = group; + break; + } + } + assertNotNull(groupWithoutAsset); + + configuredAssetValue.setAssetReferences(null); + final Map propertyValuesWithoutAsset = groupWithoutAsset.getPropertyValues(); + propertyValuesWithoutAsset.put(propertyName, configuredAssetValue); + + final ConfigurationStepEntity configurationStepEntityAfterRemoval = connectorClient.updateConfigurationStep(configurationStepEntityWithoutAsset); + assertNotNull(configurationStepEntityAfterRemoval); + + // Apply the connector update so that unreferenced assets are cleaned up based on the active configuration + final ConnectorEntity connectorBeforeApply = connectorClient.getConnector(connectorId); + assertNotNull(connectorBeforeApply); + connectorBeforeApply.setDisconnectedNodeAcknowledged(true); + + final ConnectorEntity connectorAfterApply = connectorClient.applyUpdate(connectorBeforeApply); + assertNotNull(connectorAfterApply); + + getClientUtil().waitForConnectorState(connectorId, ConnectorState.STOPPED); + + // Verify that the Asset has been removed from the Connector's Assets list + final AssetsEntity assetsAfterRemoval = connectorClient.getAssets(connectorId); + assertNotNull(assetsAfterRemoval); + + final boolean assetStillPresent = assetsAfterRemoval.getAssets() != null && assetsAfterRemoval.getAssets().stream() + .filter(a -> a.getAsset() != null) + .anyMatch(a -> uploadedAssetId.equals(a.getAsset().getId())); + + assertFalse(assetStillPresent); + + // Wait for Connector to stop before attempting to delete it. + getClientUtil().waitForConnectorStopped(connectorAfterApply.getId()); + connectorClient.deleteConnector(connectorAfterApply); + } +} + + diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorCrudIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorCrudIT.java new file mode 100644 index 000000000000..3dec256e24f5 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorCrudIT.java @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import jakarta.ws.rs.NotFoundException; +import org.apache.nifi.components.ConfigVerificationResult.Outcome; +import org.apache.nifi.components.connector.BundleCompatibility; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.BundleDTO; +import org.apache.nifi.web.api.dto.ConfigVerificationResultDTO; +import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.dto.ProcessorDTO; +import org.apache.nifi.web.api.dto.flow.FlowDTO; +import org.apache.nifi.web.api.dto.flow.ProcessGroupFlowDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ParameterContextsEntity; +import org.apache.nifi.web.api.entity.ParameterProviderEntity; +import org.apache.nifi.web.api.entity.ProcessGroupEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessorEntity; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +public class ConnectorCrudIT extends NiFiSystemIT { + + @Test + public void testCreateConfigureRestart() throws NiFiClientException, IOException, InterruptedException { + // Create Connector + final ConnectorEntity connector = getClientUtil().createConnector("NopConnector"); + assertNotNull(connector); + + // Ensure that Parameter Context is not created for the Connector + final ParameterContextsEntity contextsEntity = getNifiClient().getParamContextClient().getParamContexts(); + assertEquals(0, contextsEntity.getParameterContexts().size()); + + // Configure the connector and apply the configuration + getClientUtil().configureConnector(connector, "Ignored Step", Map.of("Ignored Property", "Hello, World!")); + getClientUtil().applyConnectorUpdate(connector); + + // Configure with a different value + getClientUtil().configureConnector(connector, "Ignored Step", Map.of("Ignored Property", "Hola, Mundo!")); + + // Restart NiFi and ensure that we have the expected values for both the active and working configurations + getNiFiInstance().stop(); + getNiFiInstance().start(); + + // If running in cluster, wait for all nodes to be connected + if (getNumberOfNodes() > 1) { + waitForAllNodesConnected(); + } + + final ConnectorEntity connectorAfterRestart = getNifiClient().getConnectorClient().getConnector(connector.getId()); + assertNotNull(connectorAfterRestart); + + final String connectorState = connectorAfterRestart.getComponent().getState(); + assertEquals(ConnectorState.STOPPED.name(), connectorState); + + final ConnectorConfigurationDTO activeConfig = connectorAfterRestart.getComponent().getActiveConfiguration(); + final Map activeProperties = activeConfig.getConfigurationStepConfigurations().getFirst().getPropertyGroupConfigurations().getFirst().getPropertyValues(); + final String activeIgnoredProperty = activeProperties.get("Ignored Property").getValue(); + assertEquals("Hello, World!", activeIgnoredProperty); + + final ConnectorConfigurationDTO workingConfig = connectorAfterRestart.getComponent().getWorkingConfiguration(); + final Map workingProperties = workingConfig.getConfigurationStepConfigurations().getFirst().getPropertyGroupConfigurations().getFirst().getPropertyValues(); + final String workingIgnoredProperty = workingProperties.get("Ignored Property").getValue(); + assertEquals("Hola, Mundo!", workingIgnoredProperty); + } + + @Test + public void testConfigVerification() throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity connector = getClientUtil().createConnector("NopConnector"); + + final List resultDtos = getClientUtil().verifyConnectorStepConfig(connector.getId(), "Ignored Step", + Map.of("Ignored Property", "Test Value")); + assertNotNull(resultDtos); + assertEquals(2, resultDtos.size()); + + assertTrue(resultDtos.stream().allMatch(result -> Outcome.SUCCESSFUL.name().equals(result.getOutcome()))); + assertEquals("Property Validation", resultDtos.getFirst().getVerificationStepName()); + + final ConfigVerificationResultDTO resultDto = resultDtos.get(1); + assertEquals("Nop Verification", resultDto.getVerificationStepName()); + assertTrue(resultDto.getExplanation().contains("Test Value")); + } + + @Test + public void testSecretReferences() throws NiFiClientException, IOException, InterruptedException { + // Create and configure a Parameter Provider with two secrets + final ParameterProviderEntity paramProvider = getClientUtil().createParameterProvider("PropertiesParameterProvider"); + getClientUtil().updateParameterProviderProperties(paramProvider, Map.of("parameters", "supersecret=supersecret\nother=other")); + + // Create the Nop Connector + final ConnectorEntity connector = getClientUtil().createConnector("NopConnector"); + assertNotNull(connector); + + // Verify that using a String Literal for a SECRET property should fail validation + final Map stringLiteralProperties = Map.of("Secret Property", "supersecret"); + List verificationResults = getClientUtil().verifyConnectorStepConfig(connector.getId(), "Ignored Step", stringLiteralProperties); + assertTrue(verificationResults.stream().anyMatch(result -> Outcome.FAILED.name().equals(result.getOutcome()))); + + // Configure and apply the String Literal value, then wait for invalid + getClientUtil().configureConnector(connector, "Ignored Step", stringLiteralProperties); + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForInvalidConnector(connector.getId()); + + // Verify that using a Secret Reference to 'other' (value "other") should fail validation + final ConnectorValueReferenceDTO otherSecretRef = getClientUtil().createSecretValueReference(paramProvider.getId(), "other", "PropertiesParameterProvider.Parameters.other"); + final Map otherSecretProperties = Map.of("Secret Property", otherSecretRef); + verificationResults = getClientUtil().verifyConnectorStepConfigWithReferences(connector.getId(), "Ignored Step", otherSecretProperties); + assertTrue(verificationResults.stream().anyMatch(result -> Outcome.FAILED.name().equals(result.getOutcome()))); + + // Configure and apply the 'other' secret reference, then wait for invalid + getClientUtil().configureConnectorWithReferences(connector.getId(), "Ignored Step", otherSecretProperties); + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForInvalidConnector(connector.getId()); + + // Verify that using an invalid reference should fail validation + final ConnectorValueReferenceDTO invalidRef = getClientUtil().createSecretValueReference(paramProvider.getId(), "nonexistent", "PropertiesParameterProvider.Parameters.nonexistent"); + final Map invalidSecretProperties = Map.of("Secret Property", invalidRef); + verificationResults = getClientUtil().verifyConnectorStepConfigWithReferences(connector.getId(), "Ignored Step", invalidSecretProperties); + assertTrue(verificationResults.stream().anyMatch(result -> Outcome.FAILED.name().equals(result.getOutcome()))); + + // Configure and apply the invalid secret reference, then wait for invalid + getClientUtil().configureConnectorWithReferences(connector.getId(), "Ignored Step", invalidSecretProperties); + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForInvalidConnector(connector.getId()); + + // Verify that using a Secret Reference to 'supersecret' (value "supersecret") should pass validation + final ConnectorValueReferenceDTO supersecretRef = getClientUtil().createSecretValueReference(paramProvider.getId(), "supersecret", "PropertiesParameterProvider.Parameters.supersecret"); + final Map supersecretProperties = Map.of("Secret Property", supersecretRef); + verificationResults = getClientUtil().verifyConnectorStepConfigWithReferences(connector.getId(), "Ignored Step", supersecretProperties); + assertTrue(verificationResults.stream().allMatch(result -> Outcome.SUCCESSFUL.name().equals(result.getOutcome()))); + + // Configure and apply the 'supersecret' secret reference, then wait for valid + getClientUtil().configureConnectorWithReferences(connector.getId(), "Ignored Step", supersecretProperties); + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForValidConnector(connector.getId()); + } + + @Test + public void testDeleteConnectorNoDataQueued() throws NiFiClientException, IOException { + // Create Connector + final ConnectorEntity connector = getClientUtil().createConnector("DataQueuingConnector"); + assertNotNull(connector); + + // Delete + getNifiClient().getConnectorClient().deleteConnector(connector); + } + + @Test + public void testProcessGroupAccessibility() throws NiFiClientException, IOException { + final ConnectorEntity connector = getClientUtil().createConnector("NestedProcessGroupConnector"); + assertNotNull(connector); + + final ConnectorEntity updatedConnector = getNifiClient().getConnectorClient().getConnector(connector.getId()); + final String managedProcessGroupId = updatedConnector.getComponent().getManagedProcessGroupId(); + assertNotNull(managedProcessGroupId); + + final ProcessGroupFlowEntity connectorFlowEntity = getNifiClient().getConnectorClient().getFlow(connector.getId(), managedProcessGroupId); + assertNotNull(connectorFlowEntity); + + final ProcessGroupFlowDTO connectorFlow = connectorFlowEntity.getProcessGroupFlow(); + assertNotNull(connectorFlow); + assertEquals(managedProcessGroupId, connectorFlow.getId()); + + try { + getNifiClient().getFlowClient().getProcessGroup(managedProcessGroupId); + fail("Was able to retrieve connector-managed process group via FlowClient"); + } catch (final NiFiClientException e) { + assertInstanceOf(NotFoundException.class, e.getCause()); + } + + final Set childGroups = connectorFlow.getFlow().getProcessGroups(); + assertEquals(1, childGroups.size(), "Expected exactly one child process group"); + + final ProcessGroupEntity childGroup = childGroups.iterator().next(); + final String childGroupId = childGroup.getId(); + assertNotNull(childGroupId); + + final ProcessGroupFlowEntity childFlowEntity = getNifiClient().getConnectorClient().getFlow(connector.getId(), childGroupId); + assertNotNull(childFlowEntity); + assertEquals(childGroupId, childFlowEntity.getProcessGroupFlow().getId()); + + try { + getNifiClient().getFlowClient().getProcessGroup(childGroupId); + fail("Was able to retrieve child process group of connector-managed flow via FlowClient"); + } catch (final NiFiClientException e) { + assertInstanceOf(NotFoundException.class, e.getCause()); + } + } + + @Test + public void testBundleResolutionRequireExactBundle() throws NiFiClientException, IOException, InterruptedException { + final ConnectorAndProcessor connectorAndProcessor = createBundleResolutionConnector(BundleCompatibility.REQUIRE_EXACT_BUNDLE); + final ProcessorDTO processor = connectorAndProcessor.processor(); + assertTrue(processor.getExtensionMissing()); + + final BundleDTO bundle = processor.getBundle(); + assertEquals("0.0.0-NONEXISTENT", bundle.getVersion()); + assertEquals(Boolean.TRUE, processor.getExtensionMissing()); + + final ConnectorEntity connector = connectorAndProcessor.connector(); + assertEquals("INVALID", connector.getComponent().getValidationStatus()); + final Collection validationErrors = connector.getComponent().getValidationErrors(); + assertNotNull(validationErrors); + assertEquals(1, validationErrors.size()); + final String validationError = validationErrors.iterator().next(); + assertTrue(validationError.contains("missing")); + } + + @Test + public void testBundleResolutionResolveBundle() throws NiFiClientException, IOException, InterruptedException { + final ProcessorDTO processor = createBundleResolutionConnector(BundleCompatibility.RESOLVE_BUNDLE).processor(); + assertFalse(processor.getExtensionMissing()); + + final BundleDTO bundle = processor.getBundle(); + assertNotEquals("0.0.0-NONEXISTENT", bundle.getVersion()); + } + + @Test + public void testBundleResolutionResolveNewestBundle() throws NiFiClientException, IOException, InterruptedException { + final ProcessorDTO processor = createBundleResolutionConnector(BundleCompatibility.RESOLVE_NEWEST_BUNDLE).processor(); + assertFalse(processor.getExtensionMissing()); + + final BundleDTO bundle = processor.getBundle(); + assertNotEquals("0.0.0-NONEXISTENT", bundle.getVersion()); + } + + private ConnectorAndProcessor createBundleResolutionConnector(final BundleCompatibility bundleCompatability) throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity connector = getClientUtil().createConnector("BundleResolutionConnector"); + assertNotNull(connector); + + getClientUtil().configureConnector(connector, "Bundle Resolution", Map.of("Bundle Compatability", bundleCompatability.name())); + getClientUtil().applyConnectorUpdate(connector); + + final ConnectorEntity updatedConnector = getNifiClient().getConnectorClient().getConnector(connector.getId()); + final ProcessGroupFlowEntity flowEntity = getNifiClient().getConnectorClient().getFlow(connector.getId()); + final FlowDTO flowDto = flowEntity.getProcessGroupFlow().getFlow(); + final Set processors = flowDto.getProcessors(); + + assertEquals(1, processors.size()); + + final ProcessorDTO processor = processors.iterator().next().getComponent(); + return new ConnectorAndProcessor(updatedConnector, processor); + } + + private record ConnectorAndProcessor(ConnectorEntity connector, ProcessorDTO processor) { + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorDrainIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorDrainIT.java new file mode 100644 index 000000000000..9f0321e87352 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorDrainIT.java @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class ConnectorDrainIT extends NiFiSystemIT { + + private static final Logger logger = LoggerFactory.getLogger(ConnectorDrainIT.class); + + @Test + public void testDrainFlowFiles() throws NiFiClientException, IOException, InterruptedException { + final File gateFile = new File(getNiFiInstance().getInstanceDirectory(), "gate-file.txt"); + gateFile.deleteOnExit(); + + if (gateFile.exists()) { + assertTrue(gateFile.delete()); + } + + logger.info("Creating GatedDataQueuingConnector"); + final ConnectorEntity connector = getClientUtil().createConnector("GatedDataQueuingConnector"); + assertNotNull(connector); + final String connectorId = connector.getId(); + + logger.info("Configuring connector {} with gate file path: {}", connectorId, gateFile.getAbsolutePath()); + getClientUtil().configureConnector(connector, "Gate Configuration", Map.of("Gate File Path", gateFile.getAbsolutePath())); + getClientUtil().applyConnectorUpdate(connector); + + logger.info("Starting connector {}", connectorId); + getClientUtil().startConnector(connectorId); + + logger.info("Waiting for at least 5 FlowFiles to queue"); + waitForConnectorMinQueueCount(connectorId, 5); + + logger.info("Stopping connector {}", connectorId); + getClientUtil().stopConnector(connectorId); + getClientUtil().waitForConnectorStopped(connectorId); + + final int queuedCountBeforeDrain = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count before drain: {}", queuedCountBeforeDrain); + assertTrue(queuedCountBeforeDrain > 0); + + logger.info("Initiating drain for connector {}", connectorId); + getClientUtil().drainConnector(connectorId); + + logger.info("Waiting for connector to enter DRAINING state"); + getClientUtil().waitForConnectorDraining(connectorId); + + logger.info("Sleeping for 2 seconds to verify connector remains in DRAINING state"); + Thread.sleep(2000L); + + ConnectorEntity drainingConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + logger.info("Connector state after 2 seconds: {}", drainingConnector.getComponent().getState()); + assertEquals(ConnectorState.DRAINING.name(), drainingConnector.getComponent().getState()); + + final int queuedWhileDraining = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count while draining (gate file absent): {}", queuedWhileDraining); + assertTrue(queuedWhileDraining > 0); + + logger.info("Creating gate file to allow draining to complete: {}", gateFile.getAbsolutePath()); + assertTrue(gateFile.createNewFile()); + + logger.info("Waiting for connector to complete draining and return to STOPPED state"); + waitFor(() -> { + try { + final ConnectorEntity entity = getNifiClient().getConnectorClient().getConnector(connectorId); + return ConnectorState.STOPPED.name().equals(entity.getComponent().getState()); + } catch (final Exception e) { + return false; + } + }); + + final ConnectorEntity finalConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + logger.info("Final connector state: {}", finalConnector.getComponent().getState()); + assertEquals(ConnectorState.STOPPED.name(), finalConnector.getComponent().getState()); + + final int finalQueuedCount = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Final queued FlowFile count: {}", finalQueuedCount); + assertEquals(0, finalQueuedCount); + + logger.info("testDrainFlowFiles completed successfully"); + } + + @Test + public void testCancelDrainFlowFiles() throws NiFiClientException, IOException, InterruptedException { + final File gateFile = new File(getNiFiInstance().getInstanceDirectory(), "gate-file-cancel.txt"); + gateFile.deleteOnExit(); + + if (gateFile.exists()) { + assertTrue(gateFile.delete()); + } + + logger.info("Creating GatedDataQueuingConnector"); + final ConnectorEntity connector = getClientUtil().createConnector("GatedDataQueuingConnector"); + assertNotNull(connector); + final String connectorId = connector.getId(); + + logger.info("Configuring connector {} with gate file path: {}", connectorId, gateFile.getAbsolutePath()); + getClientUtil().configureConnector(connector, "Gate Configuration", Map.of("Gate File Path", gateFile.getAbsolutePath())); + getClientUtil().applyConnectorUpdate(connector); + + logger.info("Starting connector {}", connectorId); + getClientUtil().startConnector(connectorId); + + logger.info("Waiting for at least 5 FlowFiles to queue"); + waitForConnectorMinQueueCount(connectorId, 5); + + logger.info("Stopping connector {}", connectorId); + getClientUtil().stopConnector(connectorId); + getClientUtil().waitForConnectorStopped(connectorId); + + final int queuedCountBeforeDrain = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count before drain: {}", queuedCountBeforeDrain); + assertTrue(queuedCountBeforeDrain > 0); + + logger.info("Initiating drain for connector {}", connectorId); + getClientUtil().drainConnector(connectorId); + + logger.info("Waiting for connector to enter DRAINING state"); + getClientUtil().waitForConnectorDraining(connectorId); + + logger.info("Sleeping for 2 seconds to verify connector remains in DRAINING state"); + Thread.sleep(2000L); + + final ConnectorEntity drainingConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + logger.info("Connector state after 2 seconds: {}", drainingConnector.getComponent().getState()); + assertEquals(ConnectorState.DRAINING.name(), drainingConnector.getComponent().getState()); + + final int queuedCountWhileDraining = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count while draining: {}", queuedCountWhileDraining); + assertTrue(queuedCountWhileDraining > 0); + + logger.info("Canceling drain for connector {}", connectorId); + getClientUtil().cancelDrain(connectorId); + + logger.info("Waiting for connector to return to STOPPED state after cancel"); + getClientUtil().waitForConnectorStopped(connectorId); + + final ConnectorEntity finalConnector = getNifiClient().getConnectorClient().getConnector(connectorId); + logger.info("Final connector state: {}", finalConnector.getComponent().getState()); + assertEquals(ConnectorState.STOPPED.name(), finalConnector.getComponent().getState()); + + final int queuedCountAfterCancel = getConnectorQueuedFlowFileCount(connectorId); + logger.info("Queued FlowFile count after cancel (should still have data): {}", queuedCountAfterCancel); + assertTrue(queuedCountAfterCancel > 0); + + logger.info("testCancelDrainFlowFiles completed successfully"); + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorLifecycleIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorLifecycleIT.java new file mode 100644 index 000000000000..dc1a50de1af3 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorLifecycleIT.java @@ -0,0 +1,235 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.groups.StatelessGroupScheduledState; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.PortDTO; +import org.apache.nifi.web.api.dto.ProcessGroupDTO; +import org.apache.nifi.web.api.dto.ProcessorDTO; +import org.apache.nifi.web.api.dto.flow.FlowDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ParameterContextEntity; +import org.apache.nifi.web.api.entity.ParameterContextsEntity; +import org.apache.nifi.web.api.entity.PortEntity; +import org.apache.nifi.web.api.entity.ProcessGroupEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessorEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Collections; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * System test that verifies the connector start/stop lifecycle correctly starts and stops + * all component types: processors, ports, and stateless groups. + */ +public class ConnectorLifecycleIT extends NiFiSystemIT { + + private static final Logger logger = LoggerFactory.getLogger(ConnectorLifecycleIT.class); + + + @Test + public void testStartStopStartsAndStopsAllComponents() throws NiFiClientException, IOException, InterruptedException { + logger.info("Creating ComponentLifecycleConnector"); + final ConnectorEntity connector = getClientUtil().createConnector("ComponentLifecycleConnector"); + assertNotNull(connector); + final String connectorId = connector.getId(); + + logger.info("Applying connector configuration"); + getClientUtil().applyConnectorUpdate(connector); + + logger.info("Waiting for connector to be valid"); + getClientUtil().waitForValidConnector(connectorId); + + logger.info("Starting connector {}", connectorId); + getClientUtil().startConnector(connectorId); + + // Ensure that there are no Parameter Contexts defined. When we start a flow that has a Stateless Group, + // we synchronize the Process Group with the Versioned External Flow, and we want to ensure that this does + // not register a Parameter Context + final ParameterContextsEntity contextsEntity = getNifiClient().getParamContextClient().getParamContexts(); + final Set parameterContexts = contextsEntity.getParameterContexts(); + assertNotNull(parameterContexts); + assertEquals(Collections.emptySet(), parameterContexts); + + logger.info("Verifying flow has components after start"); + verifyFlowHasComponents(connectorId); + + logger.info("Verifying all processors are running"); + waitForAllProcessorsRunning(connectorId); + + logger.info("Verifying all ports are running"); + waitForAllPortsRunning(connectorId); + + logger.info("Verifying stateless group is running"); + waitForStatelessGroupRunning(connectorId); + + logger.info("Stopping connector {}", connectorId); + getClientUtil().stopConnector(connectorId); + getClientUtil().waitForConnectorStopped(connectorId); + + logger.info("Verifying all processors are stopped"); + waitForAllProcessorsStopped(connectorId); + + logger.info("Verifying all ports are stopped"); + waitForAllPortsStopped(connectorId); + + logger.info("Verifying stateless group is stopped"); + waitForStatelessGroupStopped(connectorId); + + logger.info("testStartStopStartsAndStopsAllComponents completed successfully"); + } + + private void verifyFlowHasComponents(final String connectorId) throws NiFiClientException, IOException { + final ProcessGroupFlowEntity flowEntity = getNifiClient().getConnectorClient().getFlow(connectorId); + final FlowDTO flowDto = flowEntity.getProcessGroupFlow().getFlow(); + + boolean hasProcessors = !flowDto.getProcessors().isEmpty(); + boolean hasChildGroups = !flowDto.getProcessGroups().isEmpty(); + + logger.info("Root group has {} processors and {} child groups", + flowDto.getProcessors().size(), flowDto.getProcessGroups().size()); + + assertTrue(hasProcessors || hasChildGroups, "Flow should have processors or child groups"); + } + + private void waitForAllProcessorsRunning(final String connectorId) throws InterruptedException { + waitFor(() -> allProcessorsInState(connectorId, null, ScheduledState.RUNNING.name())); + } + + private void waitForAllProcessorsStopped(final String connectorId) throws InterruptedException { + waitFor(() -> allProcessorsInState(connectorId, null, ScheduledState.STOPPED.name())); + } + + private boolean allProcessorsInState(final String connectorId, final String groupId, final String expectedState) throws NiFiClientException, IOException { + final FlowDTO flowDto = getFlow(connectorId, groupId); + + for (final ProcessorEntity processorEntity : flowDto.getProcessors()) { + final ProcessorDTO processorDto = processorEntity.getComponent(); + final String state = processorDto.getState(); + if (!expectedState.equals(state) && !ScheduledState.DISABLED.name().equals(state)) { + logger.debug("Processor {} ({}) in group {} is in state {} (expected {})", + processorDto.getName(), processorEntity.getId(), groupId, state, expectedState); + return false; + } + } + + for (final ProcessGroupEntity childGroupEntity : flowDto.getProcessGroups()) { + final ProcessGroupDTO childGroupDto = childGroupEntity.getComponent(); + if (!"STATELESS".equals(childGroupDto.getExecutionEngine())) { + if (!allProcessorsInState(connectorId, childGroupEntity.getId(), expectedState)) { + return false; + } + } + } + + return true; + } + + private void waitForAllPortsRunning(final String connectorId) throws InterruptedException { + waitFor(() -> allPortsInState(connectorId, null, ScheduledState.RUNNING.name())); + } + + private void waitForAllPortsStopped(final String connectorId) throws InterruptedException { + waitFor(() -> allPortsInState(connectorId, null, ScheduledState.STOPPED.name())); + } + + private boolean allPortsInState(final String connectorId, final String groupId, final String expectedState) throws NiFiClientException, IOException { + final FlowDTO flowDto = getFlow(connectorId, groupId); + + for (final PortEntity portEntity : flowDto.getInputPorts()) { + final PortDTO portDto = portEntity.getComponent(); + final String state = portDto.getState(); + if (!expectedState.equals(state) && !ScheduledState.DISABLED.name().equals(state)) { + logger.debug("Input port {} is in state {} (expected {})", portDto.getName(), state, expectedState); + return false; + } + } + + for (final PortEntity portEntity : flowDto.getOutputPorts()) { + final PortDTO portDto = portEntity.getComponent(); + final String state = portDto.getState(); + if (!expectedState.equals(state) && !ScheduledState.DISABLED.name().equals(state)) { + logger.debug("Output port {} is in state {} (expected {})", portDto.getName(), state, expectedState); + return false; + } + } + + for (final ProcessGroupEntity childGroupEntity : flowDto.getProcessGroups()) { + final ProcessGroupDTO childGroupDto = childGroupEntity.getComponent(); + if (!"STATELESS".equals(childGroupDto.getExecutionEngine())) { + if (!allPortsInState(connectorId, childGroupEntity.getId(), expectedState)) { + return false; + } + } + } + + return true; + } + + private void waitForStatelessGroupRunning(final String connectorId) throws InterruptedException { + waitFor(() -> isStatelessGroupInState(connectorId, StatelessGroupScheduledState.RUNNING.name())); + } + + private void waitForStatelessGroupStopped(final String connectorId) throws InterruptedException { + waitFor(() -> isStatelessGroupInState(connectorId, StatelessGroupScheduledState.STOPPED.name())); + } + + private boolean isStatelessGroupInState(final String connectorId, final String expectedState) throws NiFiClientException, IOException { + return findStatelessGroupInState(connectorId, null, expectedState); + } + + private boolean findStatelessGroupInState(final String connectorId, final String groupId, final String expectedState) throws NiFiClientException, IOException { + final FlowDTO flowDto = getFlow(connectorId, groupId); + + for (final ProcessGroupEntity childGroupEntity : flowDto.getProcessGroups()) { + final ProcessGroupDTO childGroupDto = childGroupEntity.getComponent(); + if ("STATELESS".equals(childGroupDto.getExecutionEngine())) { + final String actualState = childGroupDto.getStatelessGroupScheduledState(); + logger.debug("Stateless group {} is in state {} (expected {})", childGroupDto.getName(), actualState, expectedState); + if (expectedState.equals(actualState)) { + return true; + } + } else { + if (findStatelessGroupInState(connectorId, childGroupEntity.getId(), expectedState)) { + return true; + } + } + } + + return false; + } + + private FlowDTO getFlow(final String connectorId, final String groupId) throws NiFiClientException, IOException { + final ProcessGroupFlowEntity flowEntity = (groupId == null) + ? getNifiClient().getConnectorClient().getFlow(connectorId) + : getNifiClient().getConnectorClient().getFlow(connectorId, groupId); + + return flowEntity.getProcessGroupFlow().getFlow(); + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorMethodMarshallingIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorMethodMarshallingIT.java new file mode 100644 index 000000000000..e57bf852d28d --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorMethodMarshallingIT.java @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * System tests that verify JSON marshalling of complex objects across ClassLoader boundaries + * when invoking ConnectorMethods on Processors. + */ +public class ConnectorMethodMarshallingIT extends NiFiSystemIT { + + @Test + public void testComplexObjectMarshalling() throws NiFiClientException, IOException, InterruptedException { + final File outputFile = new File("target/calculate-result.txt"); + if (outputFile.exists()) { + assertTrue(outputFile.delete(), "Failed to delete existing output file"); + } + + final ConnectorEntity connector = getClientUtil().createConnector("CalculateConnector"); + assertNotNull(connector); + + getClientUtil().configureConnector(connector, "Calculation", Map.of( + "Operand 1", "10", + "Operand 2", "5", + "Operation", "ADD", + "Output File", outputFile.getAbsolutePath() + )); + + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForValidConnector(connector.getId()); + + assertTrue(outputFile.exists(), "Output file was not created"); + final String fileContents = Files.readString(outputFile.toPath(), StandardCharsets.UTF_8); + final String[] lines = fileContents.split("\n"); + assertEquals(4, lines.length, "Output file should contain 4 lines"); + assertEquals("10", lines[0], "First line should be operand1"); + assertEquals("5", lines[1], "Second line should be operand2"); + assertEquals("ADD", lines[2], "Third line should be the operation"); + assertEquals("15", lines[3], "Fourth line should be the result (10 + 5 = 15)"); + } +} + diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorParameterContextIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorParameterContextIT.java new file mode 100644 index 000000000000..40b46eeacb0d --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorParameterContextIT.java @@ -0,0 +1,119 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.ConnectorClient; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.AssetReferenceDTO; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ParameterProviderEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * System test that validates Connectors can use Parameter Contexts with inheritance, + * sensitive parameters (via SECRET_REFERENCE), and asset-referencing parameters. + */ +public class ConnectorParameterContextIT extends NiFiSystemIT { + private static final Logger LOGGER = LoggerFactory.getLogger(ConnectorParameterContextIT.class); + private static final String SENSITIVE_SECRET_VALUE = "my-super-secret-value"; + private static final String ASSET_FILE_CONTENT = "Hello, World!"; + + @Test + public void testParameterContextInheritanceWithSensitiveAndAssetParameters() throws NiFiClientException, IOException, InterruptedException { + final File sensitiveOutputFile = new File("target/sensitive.txt"); + final File assetOutputFile = new File("target/asset.txt"); + + sensitiveOutputFile.delete(); + assetOutputFile.delete(); + + final ParameterProviderEntity paramProvider = getClientUtil().createParameterProvider("PropertiesParameterProvider"); + getClientUtil().updateParameterProviderProperties(paramProvider, Map.of("parameters", "secret=" + SENSITIVE_SECRET_VALUE)); + + final ConnectorEntity connector = getClientUtil().createConnector("ParameterContextConnector"); + assertNotNull(connector); + assertNotNull(connector.getId()); + + final String connectorId = connector.getId(); + final ConnectorClient connectorClient = getNifiClient().getConnectorClient(); + + final File assetFile = new File("src/test/resources/sample-assets/helloworld.txt"); + final AssetEntity assetEntity = connectorClient.createAsset(connectorId, assetFile.getName(), assetFile); + assertNotNull(assetEntity); + assertNotNull(assetEntity.getAsset()); + assertNotNull(assetEntity.getAsset().getId()); + + final String uploadedAssetId = assetEntity.getAsset().getId(); + + final ConnectorValueReferenceDTO secretRef = getClientUtil().createSecretValueReference( + paramProvider.getId(), "secret", "PropertiesParameterProvider.Parameters.secret"); + + final ConnectorValueReferenceDTO assetRef = new ConnectorValueReferenceDTO(); + assetRef.setValueType("ASSET_REFERENCE"); + assetRef.setAssetReferences(List.of(new AssetReferenceDTO(uploadedAssetId))); + + final Map propertyValues = new HashMap<>(); + propertyValues.put("Sensitive Value", secretRef); + propertyValues.put("Asset File", assetRef); + propertyValues.put("Sensitive Output File", createStringLiteralRef(sensitiveOutputFile.getAbsolutePath())); + propertyValues.put("Asset Output File", createStringLiteralRef(assetOutputFile.getAbsolutePath())); + + getClientUtil().configureConnectorWithReferences(connectorId, "Parameter Context Configuration", propertyValues); + LOGGER.info("Applying configuration to Connector..."); + getClientUtil().applyConnectorUpdate(connector); + LOGGER.info("Waiting for Connector to become valid..."); + getClientUtil().waitForValidConnector(connectorId); + + LOGGER.info("Connector is valid; starting Connector..."); + getClientUtil().startConnector(connectorId); + + LOGGER.info("Waiting for output files to be created..."); + waitFor(() -> sensitiveOutputFile.exists() && assetOutputFile.exists()); + + assertTrue(sensitiveOutputFile.exists(), "Sensitive output file should exist"); + assertTrue(assetOutputFile.exists(), "Asset output file should exist"); + + final String sensitiveContent = Files.readString(sensitiveOutputFile.toPath()).trim(); + final String assetContent = Files.readString(assetOutputFile.toPath()).trim(); + + assertEquals(SENSITIVE_SECRET_VALUE, sensitiveContent, "Sensitive output file should contain the secret value"); + assertEquals(ASSET_FILE_CONTENT, assetContent, "Asset output file should contain the asset file contents"); + } + + private ConnectorValueReferenceDTO createStringLiteralRef(final String value) { + final ConnectorValueReferenceDTO ref = new ConnectorValueReferenceDTO(); + ref.setValueType("STRING_LITERAL"); + ref.setValue(value); + return ref; + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorProcessorValidationIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorProcessorValidationIT.java new file mode 100644 index 000000000000..e140d883ca3f --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorProcessorValidationIT.java @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.controller.ScheduledState; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.ProcessorDTO; +import org.apache.nifi.web.api.dto.flow.FlowDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ProcessGroupEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessorEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * Verifies that processors within a Connector's managed ProcessGroup complete validation + * after a NiFi restart. Reproduces the issue where connector-internal processors get stuck + * in VALIDATING state because TriggerValidationTask only walks the root group hierarchy, + * and connector-managed groups are not descendants of the root group. + */ +public class ConnectorProcessorValidationIT extends NiFiSystemIT { + + private static final Logger logger = LoggerFactory.getLogger(ConnectorProcessorValidationIT.class); + + @Test + public void testStoppedConnectorProcessorsValidateAfterRestart() throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity connector = getClientUtil().createConnector("ComponentLifecycleConnector"); + final String connectorId = connector.getId(); + + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForValidConnector(connectorId); + waitForAllConnectorProcessorsValidated(connectorId); + logger.info("All connector processors validated before restart"); + + getNiFiInstance().stop(); + getNiFiInstance().start(); + + getClientUtil().waitForValidConnector(connectorId); + waitForAllConnectorProcessorsValidated(connectorId); + logger.info("All connector processors validated after restart"); + } + + private void waitForAllConnectorProcessorsValidated(final String connectorId) throws InterruptedException { + waitFor(() -> { + try { + final List stuckProcessors = getValidatingProcessors(connectorId, null); + if (!stuckProcessors.isEmpty()) { + logger.debug("{} connector processor(s) still in VALIDATING state: {}", + stuckProcessors.size(), stuckProcessors.stream().map(ProcessorDTO::getName).toList()); + return false; + } + + return true; + } catch (final Exception e) { + logger.debug("Failed to retrieve connector flow for validation check", e); + return false; + } + }); + } + + private List getValidatingProcessors(final String connectorId, final String groupId) throws NiFiClientException, IOException { + final ProcessGroupFlowEntity flowEntity = (groupId == null) + ? getNifiClient().getConnectorClient().getFlow(connectorId) + : getNifiClient().getConnectorClient().getFlow(connectorId, groupId); + + final FlowDTO flowDto = flowEntity.getProcessGroupFlow().getFlow(); + final List matching = new ArrayList<>(); + + for (final ProcessorEntity processorEntity : flowDto.getProcessors()) { + final ProcessorDTO dto = processorEntity.getComponent(); + if (ScheduledState.DISABLED.name().equals(dto.getState())) { + continue; + } + + if (ProcessorDTO.VALIDATING.equalsIgnoreCase(dto.getValidationStatus())) { + matching.add(dto); + } + } + + for (final ProcessGroupEntity childGroupEntity : flowDto.getProcessGroups()) { + matching.addAll(getValidatingProcessors(connectorId, childGroupEntity.getId())); + } + + return matching; + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorVersionResolutionIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorVersionResolutionIT.java new file mode 100644 index 000000000000..2c10e8093d6e --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorVersionResolutionIT.java @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.stream.io.GZIPOutputStream; +import org.apache.nifi.tests.system.InstanceConfiguration; +import org.apache.nifi.tests.system.NiFiInstanceFactory; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.tests.system.SpawnedStandaloneNiFiInstanceFactory; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.BundleDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; + +/** + * System test that verifies that when a flow.json.gz contains a Connector whose + * bundle version does not exist, but only one version of that Connector type is + * available, the Connector is created with the correct/available version. + */ +class ConnectorVersionResolutionIT extends NiFiSystemIT { + private static final String CONNECTOR_ID = "11111111-1111-1111-1111-111111111111"; + private static final String NONEXISTENT_VERSION = "1.0.0-nonexistent"; + + @Override + public NiFiInstanceFactory getInstanceFactory() { + final Path flowJsonInputPath = Paths.get("src/test/resources/flows/connector-version-mismatch/flow.json"); + final Path flowJsonOutputPath = Paths.get("target/connector-version-mismatch-flow.json.gz").toAbsolutePath(); + + try (final InputStream inputStream = Files.newInputStream(flowJsonInputPath); + final OutputStream outputStream = new GZIPOutputStream(Files.newOutputStream(flowJsonOutputPath))) { + inputStream.transferTo(outputStream); + } catch (final IOException e) { + throw new UncheckedIOException("Failed to compress Flow Configuration [%s]".formatted(flowJsonInputPath), e); + } + + return new SpawnedStandaloneNiFiInstanceFactory( + new InstanceConfiguration.Builder() + .bootstrapConfig("src/test/resources/conf/default/bootstrap.conf") + .instanceDirectory("target/standalone-instance") + .flowJson(flowJsonOutputPath.toFile()) + .overrideNifiProperties(getNifiPropertiesOverrides()) + .build() + ); + } + + @Override + protected boolean isAllowFactoryReuse() { + return false; + } + + @Override + protected boolean isDestroyEnvironmentAfterEachTest() { + return true; + } + + @Test + void testConnectorCreatedWithCorrectVersionWhenOnlyOneVersionExists() throws NiFiClientException, IOException { + final ConnectorEntity connector = getNifiClient().getConnectorClient().getConnector(CONNECTOR_ID); + assertNotNull(connector); + assertNotNull(connector.getComponent()); + + final BundleDTO bundle = connector.getComponent().getBundle(); + assertNotNull(bundle, "Connector bundle should not be null"); + assertEquals("org.apache.nifi", bundle.getGroup()); + assertEquals("nifi-system-test-extensions-nar", bundle.getArtifact()); + + assertNotEquals(NONEXISTENT_VERSION, bundle.getVersion(), "Connector should not have the nonexistent version from flow.json"); + assertEquals(getNiFiVersion(), bundle.getVersion(), "Connector should be created with the current NiFi version since only one version exists"); + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/SelectiveDropConnectorIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/SelectiveDropConnectorIT.java new file mode 100644 index 000000000000..7bdff5b6aa99 --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/SelectiveDropConnectorIT.java @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.tests.system.InstanceConfiguration; +import org.apache.nifi.tests.system.NiFiInstanceFactory; +import org.apache.nifi.tests.system.NiFiSystemIT; +import org.apache.nifi.tests.system.SpawnedStandaloneNiFiInstanceFactory; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.flow.ProcessGroupFlowDTO; +import org.apache.nifi.web.api.entity.ConnectionEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.util.Map; +import java.util.Set; +import java.util.function.Predicate; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; + +/** + * System test for the SelectiveDropConnector which tests the ability to selectively drop FlowFiles + * based on a predicate when a Connector is stopped. + */ +public class SelectiveDropConnectorIT extends NiFiSystemIT { + private static final Logger logger = LoggerFactory.getLogger(SelectiveDropConnectorIT.class); + + @Override + public NiFiInstanceFactory getInstanceFactory() { + // Ensure that each FlowFile is written to its own content claim by setting max appendable size to 1 byte + // and set the swap threshold to 2,000 FlowFiles to allow for multiple swap files to be created during the test + final Map nifiPropertiesOverrides = Map.of( + "nifi.content.claim.max.appendable.size", "1 B", + "nifi.queue.swap.threshold", "2000" + ); + + return new SpawnedStandaloneNiFiInstanceFactory( + new InstanceConfiguration.Builder() + .bootstrapConfig("src/test/resources/conf/default/bootstrap.conf") + .instanceDirectory("target/standalone-instance") + .overrideNifiProperties(nifiPropertiesOverrides) + .build()); + } + + // Ensure environment is destroyed after each test to reset nifi.properties overrides + @Override + protected boolean isDestroyEnvironmentAfterEachTest() { + return true; + } + + // Do not allow factory reuse as the test modifies nifi.properties + @Override + protected boolean isAllowFactoryReuse() { + return false; + } + + @Test + public void testSelectiveDropOnConnectorStop() throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity connector = getClientUtil().createConnector("SelectiveDropConnector"); + assertNotNull(connector); + + getNifiClient().getConnectorClient().startConnector(connector); + logger.info("Started SelectiveDropConnector, waiting for 20,000 FlowFiles to be queued"); + + waitForConnectorQueueSize(connector.getId(), 20000); + logger.info("Queue has reached 20,000 FlowFiles"); + + // We expect 9 swap files because it should be 2,000 FlowFiles in active queue and 18,000 FlowFiles swapped out. + // With 2,000 per swap file, that results in 9 swap files. + final File instanceDir = getNiFiInstance().getInstanceDirectory(); + waitForSwapFileCount(9); + + getClientUtil().stopConnector(connector); + logger.info("Stopped SelectiveDropConnector, which should have triggered selective drop of even-indexed FlowFiles"); + + final int queuedFlowFiles = getConnectorQueueSize(connector.getId()); + assertEquals(10000, queuedFlowFiles); + + final int swapFileCountAfterPurge = countSwapFiles(instanceDir); + assertEquals(9, swapFileCountAfterPurge); + + logger.info("Restarting NiFi instance to verify state persists across restart"); + getNiFiInstance().stop(); + getNiFiInstance().start(); + + final int queuedFlowFilesAfterRestart = getConnectorQueueSize(connector.getId()); + assertEquals(10000, queuedFlowFilesAfterRestart); + + final int contentFileCountWithArchive = countContentRepositoryFiles(instanceDir, true); + final int contentFileCountWithoutArchive = countContentRepositoryFiles(instanceDir, false); + assertEquals(10000, contentFileCountWithoutArchive); + assertEquals(20000, contentFileCountWithArchive); + + final int swapFileCountAfterRestart = countSwapFiles(instanceDir); + assertEquals(9, swapFileCountAfterRestart); + } + + private void waitForSwapFileCount(final int expectedCount) throws InterruptedException { + waitFor(() -> { + final File instanceDir = getNiFiInstance().getInstanceDirectory(); + final int swapFileCount = countSwapFiles(instanceDir); + return swapFileCount == expectedCount; + }); + } + + private void waitForConnectorQueueSize(final String connectorId, final int expectedSize) throws InterruptedException { + waitFor(() -> getConnectorQueueSize(connectorId) >= expectedSize); + } + + private int getConnectorQueueSize(final String connectorId) throws NiFiClientException, IOException { + final ConnectorEntity connector = getNifiClient().getConnectorClient().getConnector(connectorId); + final String managedProcessGroupId = connector.getComponent().getManagedProcessGroupId(); + if (managedProcessGroupId == null) { + return 0; + } + + final ProcessGroupFlowEntity flowEntity = getNifiClient().getConnectorClient().getFlow(connectorId, managedProcessGroupId); + final ProcessGroupFlowDTO flowDto = flowEntity.getProcessGroupFlow(); + if (flowDto == null || flowDto.getFlow() == null) { + return 0; + } + + final Set connections = flowDto.getFlow().getConnections(); + int totalQueueSize = 0; + for (final ConnectionEntity connection : connections) { + if (connection.getStatus() != null && connection.getStatus().getAggregateSnapshot() != null) { + totalQueueSize += connection.getStatus().getAggregateSnapshot().getFlowFilesQueued(); + } + } + return totalQueueSize; + } + + private int countContentRepositoryFiles(final File instanceDir, final boolean includeArchive) { + final File contentRepo = new File(instanceDir, "content_repository"); + final Predicate filter = includeArchive ? file -> true : file -> !file.getName().equals("archive"); + return countFilesRecursively(contentRepo, filter); + } + + private int countSwapFiles(final File instanceDir) { + final File flowFileRepo = new File(instanceDir, "flowfile_repository"); + final File swapDir = new File(flowFileRepo, "swap"); + final File[] swapFiles = swapDir.listFiles(); + return swapFiles == null ? 0 : swapFiles.length; + } + + private int countFilesRecursively(final File directory, final Predicate filter) { + if (!directory.exists() || !directory.isDirectory()) { + return 0; + } + + int count = 0; + final File[] files = directory.listFiles(); + if (files != null) { + for (final File file : files) { + if (file.isDirectory()) { + if (filter.test(file)) { + count += countFilesRecursively(file, filter); + } + } else if (filter.test(file)) { + count++; + } + } + } + + return count; + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/connector-version-mismatch/flow.json b/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/connector-version-mismatch/flow.json new file mode 100644 index 000000000000..d01e6130007e --- /dev/null +++ b/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/connector-version-mismatch/flow.json @@ -0,0 +1,52 @@ +{ + "encodingVersion": { + "majorVersion": 2, + "minorVersion": 0 + }, + "maxTimerDrivenThreadCount": 10, + "registries": [], + "parameterContexts": [], + "parameterProviders": [], + "controllerServices": [], + "reportingTasks": [], + "flowAnalysisRules": [], + "connectors": [ + { + "instanceIdentifier": "11111111-1111-1111-1111-111111111111", + "name": "Test NopConnector", + "type": "org.apache.nifi.connectors.tests.system.NopConnector", + "bundle": { + "group": "org.apache.nifi", + "artifact": "nifi-system-test-extensions-nar", + "version": "1.0.0-nonexistent" + }, + "scheduledState": "ENABLED", + "activeFlowConfiguration": [], + "workingFlowConfiguration": [] + } + ], + "rootGroup": { + "identifier": "root-group-identifier", + "instanceIdentifier": "root-group-instance-id", + "name": "NiFi Flow", + "comments": "", + "position": { + "x": 0.0, + "y": 0.0 + }, + "processGroups": [], + "remoteProcessGroups": [], + "processors": [], + "inputPorts": [], + "outputPorts": [], + "connections": [], + "labels": [], + "funnels": [], + "controllerServices": [], + "defaultFlowFileExpiration": "0 sec", + "defaultBackPressureObjectThreshold": 10000, + "defaultBackPressureDataSizeThreshold": "1 GB", + "flowFileConcurrency": "UNBOUNDED", + "flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE" + } +} diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/controller-service-enabling-failure/flow.json b/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/controller-service-enabling-failure/flow.json index 32dd9ba211db..08d40b7a7795 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/controller-service-enabling-failure/flow.json +++ b/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/controller-service-enabling-failure/flow.json @@ -47,7 +47,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "Enable Failure Count": "1000" diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/migrate-properties/flow.json b/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/migrate-properties/flow.json index 69ba4d539965..0657fb2429f5 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/migrate-properties/flow.json +++ b/nifi-system-tests/nifi-system-test-suite/src/test/resources/flows/migrate-properties/flow.json @@ -45,7 +45,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "count-service": "c1a97022-019a-1000-4c72-79bb12714423" @@ -82,7 +82,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "count-service": "c1a97022-019a-1000-4c72-79bb12714423" @@ -121,7 +121,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "count-service": "c1a97022-019a-1000-4c72-79bb12714423" @@ -158,7 +158,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "count-service": "c1a97022-019a-1000-4c72-79bb12714423" @@ -307,7 +307,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "7" @@ -319,7 +319,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -337,7 +337,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "9" @@ -349,7 +349,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -367,7 +367,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "6" @@ -379,7 +379,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -397,7 +397,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "10" @@ -409,7 +409,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -427,7 +427,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "1" @@ -439,7 +439,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -457,7 +457,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "0", @@ -470,7 +470,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -488,7 +488,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "11" @@ -500,7 +500,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -518,7 +518,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "3" @@ -530,7 +530,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -548,7 +548,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "5" @@ -560,7 +560,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -578,7 +578,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "4" @@ -590,7 +590,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -608,7 +608,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "2" @@ -620,7 +620,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], @@ -638,7 +638,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" }, "properties": { "start-value": "8" @@ -650,7 +650,7 @@ "bundle": { "group": "org.apache.nifi", "artifact": "nifi-system-test-extensions-services-api-nar", - "version": "2.7.0-SNAPSHOT" + "version": "2.8.0-SNAPSHOT" } } ], diff --git a/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/NiFiClientFactory.java b/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/NiFiClientFactory.java index 3c9d439f67cc..c33ca2fa6612 100644 --- a/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/NiFiClientFactory.java +++ b/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/NiFiClientFactory.java @@ -23,6 +23,7 @@ import org.apache.nifi.toolkit.cli.impl.command.CommandOption; import org.apache.nifi.toolkit.client.AccessClient; import org.apache.nifi.toolkit.client.ConnectionClient; +import org.apache.nifi.toolkit.client.ConnectorClient; import org.apache.nifi.toolkit.client.ControllerClient; import org.apache.nifi.toolkit.client.ControllerServicesClient; import org.apache.nifi.toolkit.client.CountersClient; @@ -412,6 +413,16 @@ public SystemDiagnosticsClient getSystemsDiagnosticsClient(final RequestConfig r return wrappedClient.getSystemsDiagnosticsClient(requestConfig); } + @Override + public ConnectorClient getConnectorClient() { + return wrappedClient.getConnectorClient(); + } + + @Override + public ConnectorClient getConnectorClient(final RequestConfig requestConfig) { + return wrappedClient.getConnectorClient(requestConfig); + } + @Override public void close() throws IOException { wrappedClient.close(); diff --git a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/ConnectorClient.java b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/ConnectorClient.java new file mode 100644 index 000000000000..05f9ed173c3d --- /dev/null +++ b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/ConnectorClient.java @@ -0,0 +1,419 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.toolkit.client; + +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.AssetsEntity; +import org.apache.nifi.web.api.entity.ComponentStateEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; +import org.apache.nifi.web.api.entity.DropRequestEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessGroupStatusEntity; +import org.apache.nifi.web.api.entity.VerifyConnectorConfigStepRequestEntity; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Path; + +/** + * Client for interacting with the Connector REST endpoints. + */ +public interface ConnectorClient { + + /** + * Creates a new connector. + * + * @param connectorEntity the connector entity to create + * @return the created connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity createConnector(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Gets a connector by ID. + * + * @param connectorId the connector ID + * @return the connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity getConnector(String connectorId) throws NiFiClientException, IOException; + + /** + * Updates a connector. + * + * @param connectorEntity the connector entity with updates + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity updateConnector(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Deletes a connector. + * + * @param connectorId the connector ID + * @param clientId the client ID + * @param version the revision version + * @return the deleted connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity deleteConnector(String connectorId, String clientId, long version) throws NiFiClientException, IOException; + + /** + * Deletes a connector using the information from the entity. + * + * @param connectorEntity the connector entity to delete + * @return the deleted connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity deleteConnector(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Starts a connector. + * + * @param connectorId the connector ID + * @param clientId the client ID + * @param version the revision version + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity startConnector(String connectorId, String clientId, long version) throws NiFiClientException, IOException; + + /** + * Starts a connector using the information from the entity. + * + * @param connectorEntity the connector entity to start + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity startConnector(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Stops a connector. + * + * @param connectorId the connector ID + * @param clientId the client ID + * @param version the revision version + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity stopConnector(String connectorId, String clientId, long version) throws NiFiClientException, IOException; + + /** + * Stops a connector using the information from the entity. + * + * @param connectorEntity the connector entity to stop + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity stopConnector(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Initiates draining of FlowFiles for a connector. + * + * @param connectorId the connector ID + * @param clientId the client ID + * @param version the revision version + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity drainConnector(String connectorId, String clientId, long version) throws NiFiClientException, IOException; + + /** + * Initiates draining of FlowFiles for a connector using the information from the entity. + * + * @param connectorEntity the connector entity to drain + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity drainConnector(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Cancels an ongoing drain operation for a connector. + * + * @param connectorId the connector ID + * @param clientId the client ID + * @param version the revision version + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity cancelDrain(String connectorId, String clientId, long version) throws NiFiClientException, IOException; + + /** + * Cancels an ongoing drain operation for a connector using the information from the entity. + * + * @param connectorEntity the connector entity to cancel draining for + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity cancelDrain(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Gets the configuration step names for a connector. + * + * @param connectorId the connector ID + * @return the configuration step names entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConfigurationStepNamesEntity getConfigurationSteps(String connectorId) throws NiFiClientException, IOException; + + /** + * Gets a specific configuration step for a connector. + * + * @param connectorId the connector ID + * @param configurationStepName the configuration step name + * @return the configuration step entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConfigurationStepEntity getConfigurationStep(String connectorId, String configurationStepName) throws NiFiClientException, IOException; + + /** + * Gets the allowable values for a property in a configuration step. + * + * @param connectorId the connector ID + * @param configurationStepName the configuration step name + * @param propertyGroupName the property group name + * @param propertyName the property name + * @param filter optional filter for the allowable values + * @return the allowable values entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorPropertyAllowableValuesEntity getPropertyAllowableValues(String connectorId, String configurationStepName, + String propertyGroupName, String propertyName, String filter) throws NiFiClientException, IOException; + + /** + * Updates a configuration step for a connector. + * + * @param configurationStepEntity the configuration step entity with updates + * @return the updated configuration step entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConfigurationStepEntity updateConfigurationStep(ConfigurationStepEntity configurationStepEntity) throws NiFiClientException, IOException; + + /** + * Submits a configuration step verification request. + * + * @param requestEntity the verification request entity + * @return the verification request entity with status + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + VerifyConnectorConfigStepRequestEntity submitConfigStepVerificationRequest(VerifyConnectorConfigStepRequestEntity requestEntity) throws NiFiClientException, IOException; + + /** + * Gets a configuration step verification request. + * + * @param connectorId the connector ID + * @param configurationStepName the configuration step name + * @param requestId the verification request ID + * @return the verification request entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + VerifyConnectorConfigStepRequestEntity getConfigStepVerificationRequest(String connectorId, String configurationStepName, + String requestId) throws NiFiClientException, IOException; + + /** + * Deletes a configuration step verification request. + * + * @param connectorId the connector ID + * @param configurationStepName the configuration step name + * @param requestId the verification request ID + * @return the deleted verification request entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + VerifyConnectorConfigStepRequestEntity deleteConfigStepVerificationRequest(String connectorId, String configurationStepName, + String requestId) throws NiFiClientException, IOException; + + /** + * Applies an update to a connector. + * + * @param connectorEntity the connector entity with revision + * @return the updated connector entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ConnectorEntity applyUpdate(ConnectorEntity connectorEntity) throws NiFiClientException, IOException; + + /** + * Gets the flow for the process group managed by a connector. + * + * @param connectorId the connector ID + * @return the process group flow entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ProcessGroupFlowEntity getFlow(String connectorId) throws NiFiClientException, IOException; + + /** + * Gets the flow for a specific process group within a connector's managed flow. + * + * @param connectorId the connector ID + * @param processGroupId the process group ID within the connector's managed flow + * @return the process group flow entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ProcessGroupFlowEntity getFlow(String connectorId, String processGroupId) throws NiFiClientException, IOException; + + /** + * Gets the status for the process group managed by a connector. + * + * @param connectorId the connector ID + * @param recursive whether to include status for all descendant components + * @return the process group status entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ProcessGroupStatusEntity getStatus(String connectorId, boolean recursive) throws NiFiClientException, IOException; + + /** + * Creates an asset in the given connector. + * + * @param connectorId the connector ID + * @param assetName the asset name + * @param file the file + * @return the created asset entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + AssetEntity createAsset(String connectorId, String assetName, File file) throws NiFiClientException, IOException; + + /** + * Lists the assets for the given connector. + * + * @param connectorId the connector ID + * @return the list of asset entities + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + AssetsEntity getAssets(String connectorId) throws NiFiClientException, IOException; + + /** + * Retrieves the content of the given asset. + * + * @param connectorId the connector ID + * @param assetId the asset ID + * @param outputDirectory the directory to write the asset content + * @return the path to the asset content + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + Path getAssetContent(String connectorId, String assetId, File outputDirectory) throws NiFiClientException, IOException; + + /** + * Creates a request to purge all FlowFiles for the given connector. + * + * @param connectorId the connector ID + * @return the drop request entity containing the purge request status + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + DropRequestEntity createPurgeRequest(String connectorId) throws NiFiClientException, IOException; + + /** + * Gets the status of a purge request for the given connector. + * + * @param connectorId the connector ID + * @param purgeRequestId the purge request ID + * @return the drop request entity containing the purge request status + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + DropRequestEntity getPurgeRequest(String connectorId, String purgeRequestId) throws NiFiClientException, IOException; + + /** + * Deletes (cancels) a purge request for the given connector. + * + * @param connectorId the connector ID + * @param purgeRequestId the purge request ID + * @return the drop request entity containing the final purge request status + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + DropRequestEntity deletePurgeRequest(String connectorId, String purgeRequestId) throws NiFiClientException, IOException; + + /** + * Gets the state for a processor within a connector. + * + * @param connectorId the connector ID + * @param processorId the processor ID + * @return the component state entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ComponentStateEntity getProcessorState(String connectorId, String processorId) throws NiFiClientException, IOException; + + /** + * Clears the state for a processor within a connector. + * + * @param connectorId the connector ID + * @param processorId the processor ID + * @return the component state entity after clearing + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ComponentStateEntity clearProcessorState(String connectorId, String processorId) throws NiFiClientException, IOException; + + /** + * Gets the state for a controller service within a connector. + * + * @param connectorId the connector ID + * @param controllerServiceId the controller service ID + * @return the component state entity + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ComponentStateEntity getControllerServiceState(String connectorId, String controllerServiceId) throws NiFiClientException, IOException; + + /** + * Clears the state for a controller service within a connector. + * + * @param connectorId the connector ID + * @param controllerServiceId the controller service ID + * @return the component state entity after clearing + * @throws NiFiClientException if an error occurs during the request + * @throws IOException if an I/O error occurs + */ + ComponentStateEntity clearControllerServiceState(String connectorId, String controllerServiceId) throws NiFiClientException, IOException; + + /** + * Indicates that mutable requests should indicate that the client has + * acknowledged that the node is disconnected. + */ + void acknowledgeDisconnectedNode(); +} diff --git a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/FlowClient.java b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/FlowClient.java index e483f1c91a58..57fa3c74f560 100644 --- a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/FlowClient.java +++ b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/FlowClient.java @@ -20,6 +20,7 @@ import org.apache.nifi.web.api.entity.ActivateControllerServicesEntity; import org.apache.nifi.web.api.entity.ClusterSummaryEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; +import org.apache.nifi.web.api.entity.ConnectorsEntity; import org.apache.nifi.web.api.entity.ControllerServiceTypesEntity; import org.apache.nifi.web.api.entity.ControllerServicesEntity; import org.apache.nifi.web.api.entity.CurrentUserEntity; @@ -229,4 +230,11 @@ VersionedFlowSnapshotMetadataSetEntity getVersions(String registryId, String buc * @return list of flows */ VersionedFlowsEntity getFlowRegistryFlows(String registryClientId, String branch, String bucket) throws NiFiClientException, IOException; + + /** + * Retrieves all of the Connectors. + * + * @return the list of Connectors + */ + ConnectorsEntity getConnectors() throws NiFiClientException, IOException; } diff --git a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/NiFiClient.java b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/NiFiClient.java index 44bc180e3aab..a70421202ac8 100644 --- a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/NiFiClient.java +++ b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/NiFiClient.java @@ -135,12 +135,18 @@ public interface NiFiClient extends Closeable { SnippetClient getSnippetClient(RequestConfig requestConfig); - // ----- SnippetClient ----- + // ----- SystemDiagnosticsClient ----- SystemDiagnosticsClient getSystemsDiagnosticsClient(); SystemDiagnosticsClient getSystemsDiagnosticsClient(RequestConfig requestConfig); + // ----- ConnectorClient ----- + + ConnectorClient getConnectorClient(); + + ConnectorClient getConnectorClient(RequestConfig requestConfig); + /** * The builder interface that implementations should provide for obtaining the * client. diff --git a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyConnectorClient.java b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyConnectorClient.java new file mode 100644 index 000000000000..7b648e7183e6 --- /dev/null +++ b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyConnectorClient.java @@ -0,0 +1,673 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.toolkit.client.impl; + +import jakarta.ws.rs.client.Entity; +import jakarta.ws.rs.client.WebTarget; +import jakarta.ws.rs.core.MediaType; +import jakarta.ws.rs.core.Response; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.toolkit.client.ConnectorClient; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.toolkit.client.RequestConfig; +import org.apache.nifi.web.api.dto.RevisionDTO; +import org.apache.nifi.web.api.entity.AssetEntity; +import org.apache.nifi.web.api.entity.AssetsEntity; +import org.apache.nifi.web.api.entity.ComponentStateEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepEntity; +import org.apache.nifi.web.api.entity.ConfigurationStepNamesEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ConnectorPropertyAllowableValuesEntity; +import org.apache.nifi.web.api.entity.ConnectorRunStatusEntity; +import org.apache.nifi.web.api.entity.DropRequestEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessGroupStatusEntity; +import org.apache.nifi.web.api.entity.VerifyConnectorConfigStepRequestEntity; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.Objects; + +/** + * Jersey implementation of ConnectorClient. + */ +public class JerseyConnectorClient extends AbstractJerseyClient implements ConnectorClient { + + private final WebTarget connectorsTarget; + private volatile WebTarget connectorTarget; + + public JerseyConnectorClient(final WebTarget baseTarget) { + this(baseTarget, null); + } + + public JerseyConnectorClient(final WebTarget baseTarget, final RequestConfig requestConfig) { + super(requestConfig); + this.connectorsTarget = baseTarget.path("/connectors"); + this.connectorTarget = baseTarget.path("/connectors/{id}"); + } + + @Override + public void acknowledgeDisconnectedNode() { + connectorTarget = connectorTarget.queryParam("disconnectedNodeAcknowledged", true); + } + + @Override + public ConnectorEntity createConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + if (connectorEntity == null) { + throw new IllegalArgumentException("Connector entity cannot be null"); + } + + return executeAction("Error creating Connector", () -> { + return getRequestBuilder(connectorsTarget).post( + Entity.entity(connectorEntity, MediaType.APPLICATION_JSON_TYPE), + ConnectorEntity.class); + }); + } + + @Override + public ConnectorEntity getConnector(final String connectorId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector ID cannot be null or blank"); + } + + return executeAction("Error retrieving Connector", () -> { + final WebTarget target = connectorTarget.resolveTemplate("id", connectorId); + return getRequestBuilder(target).get(ConnectorEntity.class); + }); + } + + @Override + public ConnectorEntity updateConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + if (connectorEntity == null) { + throw new IllegalArgumentException("Connector entity cannot be null"); + } + + return executeAction("Error updating Connector", () -> { + final WebTarget target = connectorTarget.resolveTemplate("id", connectorEntity.getId()); + return getRequestBuilder(target).put( + Entity.entity(connectorEntity, MediaType.APPLICATION_JSON_TYPE), + ConnectorEntity.class); + }); + } + + @Override + public ConnectorEntity deleteConnector(final String connectorId, final String clientId, final long version) throws NiFiClientException, IOException { + return deleteConnector(connectorId, clientId, version, false); + } + + private ConnectorEntity deleteConnector(final String connectorId, final String clientId, final long version, + final Boolean acknowledgeDisconnect) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector ID cannot be null or blank"); + } + + return executeAction("Error deleting Connector", () -> { + WebTarget target = connectorTarget + .queryParam("version", version) + .queryParam("clientId", clientId) + .resolveTemplate("id", connectorId); + + if (acknowledgeDisconnect == Boolean.TRUE) { + target = target.queryParam("disconnectedNodeAcknowledged", "true"); + } + + return getRequestBuilder(target).delete(ConnectorEntity.class); + }); + } + + @Override + public ConnectorEntity deleteConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + if (connectorEntity == null) { + throw new IllegalArgumentException("Connector entity cannot be null"); + } + return deleteConnector(connectorEntity.getId(), connectorEntity.getRevision().getClientId(), + connectorEntity.getRevision().getVersion(), connectorEntity.isDisconnectedNodeAcknowledged()); + } + + @Override + public ConnectorEntity startConnector(final String connectorId, final String clientId, final long version) throws NiFiClientException, IOException { + return updateConnectorRunStatus(connectorId, "RUNNING", clientId, version, false); + } + + @Override + public ConnectorEntity startConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + return updateConnectorRunStatus(connectorEntity.getId(), "RUNNING", connectorEntity.getRevision().getClientId(), + connectorEntity.getRevision().getVersion(), connectorEntity.isDisconnectedNodeAcknowledged()); + } + + @Override + public ConnectorEntity stopConnector(final String connectorId, final String clientId, final long version) throws NiFiClientException, IOException { + return updateConnectorRunStatus(connectorId, "STOPPED", clientId, version, false); + } + + @Override + public ConnectorEntity stopConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + return updateConnectorRunStatus(connectorEntity.getId(), "STOPPED", connectorEntity.getRevision().getClientId(), + connectorEntity.getRevision().getVersion(), connectorEntity.isDisconnectedNodeAcknowledged()); + } + + @Override + public ConnectorEntity drainConnector(final String connectorId, final String clientId, final long version) throws NiFiClientException, IOException { + return drainConnector(connectorId, clientId, version, false); + } + + @Override + public ConnectorEntity drainConnector(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + return drainConnector(connectorEntity.getId(), connectorEntity.getRevision().getClientId(), + connectorEntity.getRevision().getVersion(), connectorEntity.isDisconnectedNodeAcknowledged()); + } + + private ConnectorEntity drainConnector(final String connectorId, final String clientId, final long version, + final Boolean disconnectedNodeAcknowledged) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector ID cannot be null or blank"); + } + + return executeAction("Error initiating connector drain", () -> { + final WebTarget target = connectorTarget + .path("/drain") + .resolveTemplate("id", connectorId); + + final ConnectorEntity requestEntity = new ConnectorEntity(); + requestEntity.setId(connectorId); + requestEntity.setDisconnectedNodeAcknowledged(disconnectedNodeAcknowledged); + + final RevisionDTO revisionDto = new RevisionDTO(); + revisionDto.setClientId(clientId); + revisionDto.setVersion(version); + requestEntity.setRevision(revisionDto); + + return getRequestBuilder(target).post( + Entity.entity(requestEntity, MediaType.APPLICATION_JSON_TYPE), + ConnectorEntity.class); + }); + } + + @Override + public ConnectorEntity cancelDrain(final String connectorId, final String clientId, final long version) throws NiFiClientException, IOException { + return cancelDrain(connectorId, clientId, version, false); + } + + @Override + public ConnectorEntity cancelDrain(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + return cancelDrain(connectorEntity.getId(), connectorEntity.getRevision().getClientId(), + connectorEntity.getRevision().getVersion(), connectorEntity.isDisconnectedNodeAcknowledged()); + } + + private ConnectorEntity cancelDrain(final String connectorId, final String clientId, final long version, + final Boolean disconnectedNodeAcknowledged) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector ID cannot be null or blank"); + } + + return executeAction("Error canceling connector drain", () -> { + WebTarget target = connectorTarget + .path("/drain") + .queryParam("version", version) + .queryParam("clientId", clientId) + .resolveTemplate("id", connectorId); + + if (disconnectedNodeAcknowledged == Boolean.TRUE) { + target = target.queryParam("disconnectedNodeAcknowledged", "true"); + } + + return getRequestBuilder(target).delete(ConnectorEntity.class); + }); + } + + private ConnectorEntity updateConnectorRunStatus(final String connectorId, final String desiredState, final String clientId, + final long version, final Boolean disconnectedNodeAcknowledged) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector ID cannot be null or blank"); + } + + return executeAction("Error updating Connector run status", () -> { + final WebTarget target = connectorTarget + .path("/run-status") + .resolveTemplate("id", connectorId); + + final ConnectorRunStatusEntity runStatusEntity = new ConnectorRunStatusEntity(); + runStatusEntity.setState(desiredState); + runStatusEntity.setDisconnectedNodeAcknowledged(disconnectedNodeAcknowledged); + + final RevisionDTO revisionDto = new RevisionDTO(); + revisionDto.setClientId(clientId); + revisionDto.setVersion(version); + runStatusEntity.setRevision(revisionDto); + + return getRequestBuilder(target).put( + Entity.entity(runStatusEntity, MediaType.APPLICATION_JSON_TYPE), + ConnectorEntity.class); + }); + } + + @Override + public ConfigurationStepNamesEntity getConfigurationSteps(final String connectorId) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + + return executeAction("Error retrieving configuration steps", () -> { + final WebTarget target = connectorTarget + .path("/configuration-steps") + .resolveTemplate("id", connectorId); + return getRequestBuilder(target).get(ConfigurationStepNamesEntity.class); + }); + } + + @Override + public ConfigurationStepEntity getConfigurationStep(final String connectorId, final String configurationStepName) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + Objects.requireNonNull(configurationStepName, "Configuration step name required"); + + return executeAction("Error retrieving configuration step", () -> { + final WebTarget target = connectorTarget + .path("/configuration-steps/{configurationStepName}") + .resolveTemplate("id", connectorId) + .resolveTemplate("configurationStepName", configurationStepName); + return getRequestBuilder(target).get(ConfigurationStepEntity.class); + }); + } + + @Override + public ConnectorPropertyAllowableValuesEntity getPropertyAllowableValues(final String connectorId, final String configurationStepName, + final String propertyGroupName, final String propertyName, final String filter) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + Objects.requireNonNull(configurationStepName, "Configuration step name required"); + Objects.requireNonNull(propertyGroupName, "Property group name required"); + Objects.requireNonNull(propertyName, "Property name required"); + + return executeAction("Error retrieving property allowable values", () -> { + WebTarget target = connectorTarget + .path("/configuration-steps/{configurationStepName}/property-groups/{propertyGroupName}/properties/{propertyName}/allowable-values") + .resolveTemplate("id", connectorId) + .resolveTemplate("configurationStepName", configurationStepName) + .resolveTemplate("propertyGroupName", propertyGroupName) + .resolveTemplate("propertyName", propertyName); + + if (filter != null) { + target = target.queryParam("filter", filter); + } + + return getRequestBuilder(target).get(ConnectorPropertyAllowableValuesEntity.class); + }); + } + + @Override + public ConfigurationStepEntity updateConfigurationStep(final ConfigurationStepEntity configurationStepEntity) throws NiFiClientException, IOException { + if (configurationStepEntity == null) { + throw new IllegalArgumentException("Configuration step entity cannot be null"); + } + if (configurationStepEntity.getParentConnectorId() == null) { + throw new IllegalArgumentException("Parent connector ID cannot be null"); + } + if (configurationStepEntity.getConfigurationStep() == null) { + throw new IllegalArgumentException("Configuration step cannot be null"); + } + if (configurationStepEntity.getConfigurationStep().getConfigurationStepName() == null) { + throw new IllegalArgumentException("Configuration step name cannot be null"); + } + + return executeAction("Error updating configuration step", () -> { + final WebTarget target = connectorTarget + .path("/configuration-steps/{configurationStepName}") + .resolveTemplate("id", configurationStepEntity.getParentConnectorId()) + .resolveTemplate("configurationStepName", configurationStepEntity.getConfigurationStep().getConfigurationStepName()); + + return getRequestBuilder(target).put( + Entity.entity(configurationStepEntity, MediaType.APPLICATION_JSON_TYPE), + ConfigurationStepEntity.class); + }); + } + + @Override + public VerifyConnectorConfigStepRequestEntity submitConfigStepVerificationRequest(final VerifyConnectorConfigStepRequestEntity requestEntity) + throws NiFiClientException, IOException { + if (requestEntity == null) { + throw new IllegalArgumentException("Verification request entity cannot be null"); + } + if (requestEntity.getRequest() == null) { + throw new IllegalArgumentException("Verification request DTO cannot be null"); + } + if (requestEntity.getRequest().getConnectorId() == null) { + throw new IllegalArgumentException("Connector ID cannot be null"); + } + if (requestEntity.getRequest().getConfigurationStepName() == null) { + throw new IllegalArgumentException("Configuration step name cannot be null"); + } + + return executeAction("Error submitting configuration step verification request", () -> { + final WebTarget target = connectorTarget + .path("/configuration-steps/{configurationStepName}/verify-config") + .resolveTemplate("id", requestEntity.getRequest().getConnectorId()) + .resolveTemplate("configurationStepName", requestEntity.getRequest().getConfigurationStepName()); + + return getRequestBuilder(target).post( + Entity.entity(requestEntity, MediaType.APPLICATION_JSON_TYPE), + VerifyConnectorConfigStepRequestEntity.class); + }); + } + + @Override + public VerifyConnectorConfigStepRequestEntity getConfigStepVerificationRequest(final String connectorId, final String configurationStepName, + final String requestId) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + Objects.requireNonNull(configurationStepName, "Configuration step name required"); + Objects.requireNonNull(requestId, "Request ID required"); + + return executeAction("Error retrieving configuration step verification request", () -> { + final WebTarget target = connectorTarget + .path("/configuration-steps/{configurationStepName}/verify-config/{requestId}") + .resolveTemplate("id", connectorId) + .resolveTemplate("configurationStepName", configurationStepName) + .resolveTemplate("requestId", requestId); + + return getRequestBuilder(target).get(VerifyConnectorConfigStepRequestEntity.class); + }); + } + + @Override + public VerifyConnectorConfigStepRequestEntity deleteConfigStepVerificationRequest(final String connectorId, final String configurationStepName, + final String requestId) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + Objects.requireNonNull(configurationStepName, "Configuration step name required"); + Objects.requireNonNull(requestId, "Request ID required"); + + return executeAction("Error deleting configuration step verification request", () -> { + final WebTarget target = connectorTarget + .path("/configuration-steps/{configurationStepName}/verify-config/{requestId}") + .resolveTemplate("id", connectorId) + .resolveTemplate("configurationStepName", configurationStepName) + .resolveTemplate("requestId", requestId); + + return getRequestBuilder(target).delete(VerifyConnectorConfigStepRequestEntity.class); + }); + } + + @Override + public ConnectorEntity applyUpdate(final ConnectorEntity connectorEntity) throws NiFiClientException, IOException { + if (connectorEntity == null) { + throw new IllegalArgumentException("Connector entity cannot be null"); + } + if (StringUtils.isBlank(connectorEntity.getId())) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (connectorEntity.getRevision() == null) { + throw new IllegalArgumentException("Revision cannot be null"); + } + + final String connectorId = connectorEntity.getId(); + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector ID cannot be null or blank"); + } + + return executeAction("Error applying connector update", () -> { + final WebTarget target = connectorTarget + .path("/apply-update") + .resolveTemplate("id", connectorId); + + return getRequestBuilder(target).post( + Entity.entity(connectorEntity, MediaType.APPLICATION_JSON_TYPE), + ConnectorEntity.class); + }); + } + + @Override + public ProcessGroupFlowEntity getFlow(final String connectorId) throws NiFiClientException, IOException { + final ConnectorEntity connector = getConnector(connectorId); + final String managedProcessGroupId = connector.getComponent().getManagedProcessGroupId(); + return getFlow(connectorId, managedProcessGroupId); + } + + @Override + public ProcessGroupFlowEntity getFlow(final String connectorId, final String processGroupId) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + Objects.requireNonNull(processGroupId, "Process Group ID required"); + + return executeAction("Error retrieving connector flow", () -> { + WebTarget target = connectorTarget + .path("/flow/process-groups/{processGroupId}") + .resolveTemplate("id", connectorId) + .resolveTemplate("processGroupId", processGroupId); + + return getRequestBuilder(target).get(ProcessGroupFlowEntity.class); + }); + } + + @Override + public ProcessGroupStatusEntity getStatus(final String connectorId, final boolean recursive) throws NiFiClientException, IOException { + Objects.requireNonNull(connectorId, "Connector ID required"); + + return executeAction("Error retrieving connector status", () -> { + WebTarget target = connectorTarget + .path("/status") + .resolveTemplate("id", connectorId); + + if (recursive) { + target = target.queryParam("recursive", "true"); + } + + return getRequestBuilder(target).get(ProcessGroupStatusEntity.class); + }); + } + + @Override + public AssetEntity createAsset(final String connectorId, final String assetName, final File file) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(assetName)) { + throw new IllegalArgumentException("Asset name cannot be null or blank"); + } + if (file == null) { + throw new IllegalArgumentException("File cannot be null"); + } + if (!file.exists()) { + throw new FileNotFoundException(file.getAbsolutePath()); + } + + try (final InputStream assetInputStream = new FileInputStream(file)) { + return executeAction("Error creating Connector Asset " + assetName + " for Connector " + connectorId, () -> { + final WebTarget target = connectorsTarget + .path("{id}/assets") + .resolveTemplate("id", connectorId); + + return getRequestBuilder(target) + .header("Filename", assetName) + .post( + Entity.entity(assetInputStream, MediaType.APPLICATION_OCTET_STREAM_TYPE), + AssetEntity.class); + }); + } + } + + @Override + public AssetsEntity getAssets(final String connectorId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + + return executeAction("Error retrieving Connector assets", () -> { + final WebTarget target = connectorsTarget + .path("{id}/assets") + .resolveTemplate("id", connectorId); + return getRequestBuilder(target).get(AssetsEntity.class); + }); + } + + @Override + public Path getAssetContent(final String connectorId, final String assetId, final File outputDirectory) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(assetId)) { + throw new IllegalArgumentException("Asset id cannot be null or blank"); + } + + return executeAction("Error getting Connector asset content", () -> { + final WebTarget target = connectorsTarget + .path("{id}/assets/{assetId}") + .resolveTemplate("id", connectorId) + .resolveTemplate("assetId", assetId); + + final Response response = getRequestBuilder(target) + .accept(MediaType.APPLICATION_OCTET_STREAM_TYPE) + .get(); + + final String filename = getContentDispositionFilename(response); + final File assetFile = new File(outputDirectory, filename); + + try (final InputStream responseInputStream = response.readEntity(InputStream.class)) { + Files.copy(responseInputStream, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING); + return assetFile.toPath(); + } + }); + } + + @Override + public DropRequestEntity createPurgeRequest(final String connectorId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + + return executeAction("Error creating purge request for Connector " + connectorId, () -> { + final WebTarget target = connectorsTarget + .path("{id}/purge-requests") + .resolveTemplate("id", connectorId); + + return getRequestBuilder(target).post(null, DropRequestEntity.class); + }); + } + + @Override + public DropRequestEntity getPurgeRequest(final String connectorId, final String purgeRequestId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(purgeRequestId)) { + throw new IllegalArgumentException("Purge request id cannot be null or blank"); + } + + return executeAction("Error getting purge request for Connector " + connectorId, () -> { + final WebTarget target = connectorsTarget + .path("{id}/purge-requests/{purgeRequestId}") + .resolveTemplate("id", connectorId) + .resolveTemplate("purgeRequestId", purgeRequestId); + + return getRequestBuilder(target).get(DropRequestEntity.class); + }); + } + + @Override + public DropRequestEntity deletePurgeRequest(final String connectorId, final String purgeRequestId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(purgeRequestId)) { + throw new IllegalArgumentException("Purge request id cannot be null or blank"); + } + + return executeAction("Error deleting purge request for Connector " + connectorId, () -> { + final WebTarget target = connectorsTarget + .path("{id}/purge-requests/{purgeRequestId}") + .resolveTemplate("id", connectorId) + .resolveTemplate("purgeRequestId", purgeRequestId); + + return getRequestBuilder(target).delete(DropRequestEntity.class); + }); + } + + @Override + public ComponentStateEntity getProcessorState(final String connectorId, final String processorId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(processorId)) { + throw new IllegalArgumentException("Processor id cannot be null or blank"); + } + + return executeAction("Error retrieving processor state for Connector " + connectorId, () -> { + final WebTarget target = connectorTarget + .path("/processors/{processorId}/state") + .resolveTemplate("id", connectorId) + .resolveTemplate("processorId", processorId); + + return getRequestBuilder(target).get(ComponentStateEntity.class); + }); + } + + @Override + public ComponentStateEntity clearProcessorState(final String connectorId, final String processorId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(processorId)) { + throw new IllegalArgumentException("Processor id cannot be null or blank"); + } + + return executeAction("Error clearing processor state for Connector " + connectorId, () -> { + final WebTarget target = connectorTarget + .path("/processors/{processorId}/state/clear-requests") + .resolveTemplate("id", connectorId) + .resolveTemplate("processorId", processorId); + + return getRequestBuilder(target).post(null, ComponentStateEntity.class); + }); + } + + @Override + public ComponentStateEntity getControllerServiceState(final String connectorId, final String controllerServiceId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(controllerServiceId)) { + throw new IllegalArgumentException("Controller service id cannot be null or blank"); + } + + return executeAction("Error retrieving controller service state for Connector " + connectorId, () -> { + final WebTarget target = connectorTarget + .path("/controller-services/{controllerServiceId}/state") + .resolveTemplate("id", connectorId) + .resolveTemplate("controllerServiceId", controllerServiceId); + + return getRequestBuilder(target).get(ComponentStateEntity.class); + }); + } + + @Override + public ComponentStateEntity clearControllerServiceState(final String connectorId, final String controllerServiceId) throws NiFiClientException, IOException { + if (StringUtils.isBlank(connectorId)) { + throw new IllegalArgumentException("Connector id cannot be null or blank"); + } + if (StringUtils.isBlank(controllerServiceId)) { + throw new IllegalArgumentException("Controller service id cannot be null or blank"); + } + + return executeAction("Error clearing controller service state for Connector " + connectorId, () -> { + final WebTarget target = connectorTarget + .path("/controller-services/{controllerServiceId}/state/clear-requests") + .resolveTemplate("id", connectorId) + .resolveTemplate("controllerServiceId", controllerServiceId); + + return getRequestBuilder(target).post(null, ComponentStateEntity.class); + }); + } +} diff --git a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyFlowClient.java b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyFlowClient.java index 114c4e01c235..f0cb9450cacf 100644 --- a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyFlowClient.java +++ b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyFlowClient.java @@ -32,6 +32,7 @@ import org.apache.nifi.web.api.entity.ClusterSummaryEntity; import org.apache.nifi.web.api.entity.ComponentEntity; import org.apache.nifi.web.api.entity.ConnectionStatusEntity; +import org.apache.nifi.web.api.entity.ConnectorsEntity; import org.apache.nifi.web.api.entity.ControllerServiceTypesEntity; import org.apache.nifi.web.api.entity.ControllerServicesEntity; import org.apache.nifi.web.api.entity.CurrentUserEntity; @@ -400,4 +401,12 @@ public VersionedFlowsEntity getFlowRegistryFlows(String registryClientId, String return getRequestBuilder(target).get(VersionedFlowsEntity.class); }); } + + @Override + public ConnectorsEntity getConnectors() throws NiFiClientException, IOException { + return executeAction("Error retrieving Connectors", () -> { + final WebTarget target = flowTarget.path("/connectors"); + return getRequestBuilder(target).get(ConnectorsEntity.class); + }); + } } diff --git a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyNiFiClient.java b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyNiFiClient.java index af5697e262c1..c99f15d9e97e 100644 --- a/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyNiFiClient.java +++ b/nifi-toolkit/nifi-toolkit-client/src/main/java/org/apache/nifi/toolkit/client/impl/JerseyNiFiClient.java @@ -26,6 +26,7 @@ import org.apache.commons.lang3.StringUtils; import org.apache.nifi.toolkit.client.AccessClient; import org.apache.nifi.toolkit.client.ConnectionClient; +import org.apache.nifi.toolkit.client.ConnectorClient; import org.apache.nifi.toolkit.client.ControllerClient; import org.apache.nifi.toolkit.client.ControllerServicesClient; import org.apache.nifi.toolkit.client.CountersClient; @@ -311,6 +312,16 @@ public SystemDiagnosticsClient getSystemsDiagnosticsClient(final RequestConfig r return new JerseySystemDiagnosticsClient(baseTarget, requestConfig); } + @Override + public ConnectorClient getConnectorClient() { + return new JerseyConnectorClient(baseTarget); + } + + @Override + public ConnectorClient getConnectorClient(final RequestConfig requestConfig) { + return new JerseyConnectorClient(baseTarget, requestConfig); + } + @Override public void close() { if (this.client != null) { @@ -351,7 +362,7 @@ private static JacksonJaxbJsonProvider jacksonJaxbJsonProvider() { JacksonJaxbJsonProvider jacksonJaxbJsonProvider = new JacksonJaxbJsonProvider(); ObjectMapper mapper = new ObjectMapper(); - mapper.setDefaultPropertyInclusion(JsonInclude.Value.construct(JsonInclude.Include.NON_NULL, JsonInclude.Include.NON_NULL)); + mapper.setDefaultPropertyInclusion(JsonInclude.Value.construct(JsonInclude.Include.NON_NULL, JsonInclude.Include.ALWAYS)); mapper.setAnnotationIntrospector(new JakartaXmlBindAnnotationIntrospector(mapper.getTypeFactory())); // Ignore unknown properties so that deployed client remain compatible with // future versions of NiFi that add new fields diff --git a/pom.xml b/pom.xml index e7e7934c82d3..e95220ea3e5a 100644 --- a/pom.xml +++ b/pom.xml @@ -32,6 +32,7 @@ nifi-bootstrap nifi-code-coverage nifi-mock + nifi-connector-mock-bundle nifi-extension-bundles nifi-extension-bom nifi-assembly @@ -44,6 +45,7 @@ nifi-registry nifi-toolkit nifi-manifest + nifi-connectors c2 https://nifi.apache.org