Clever Geek Handbook

NiFi da Beauty: HTTPS / LDAP / NiFi Registry / NiFi Cli + CI / CD

Introdução

Deixe-me explicar meu acordeão imediatamente. Sim, a internet está cheia de manuais. Sim, cheio de orientações baseadas em turnos. Sim, você pode dizer que tudo é mastigado mastigado. Mas especificamente no meu caso, como sempre acontece, houve um punhado de "mas":





Existem manuais sobre como configurar um pacote NiFi e NiFi Registry com autenticação e autorização ativadas. Mas ... certificados autoassinados são usados.





Existem manuais separados sobre como aparafusar um certificado comercial para NiFi; respectivamente para o NiFi Registry "kagbe is the same". Mas a autenticação e autorização mútuas ocorrerão com o uso de SSL de duas vias ... e nós temos o LDAP ... e então você não será capaz de garantir a coerência do doce casal usando apenas o diretório externo em sua intuição pura.





Existem manuais para conexão com o LDAP para NiFi e NiFi Registry. Nãããão ... como no "mas" anterior, surgem dúvidas, como passar só com LDAP, porque ainda temos NiFi Cli, mas ele não sabe fazer LDAP.





Em outras palavras, todos os manuais têm uma pequena nuance: eles cobrem apenas os cenários mais simples. Simplesmente não há documentação para pacotes complexos. Além disso, no decorrer da configuração do pacote, encontrei dificuldades que são encontradas apenas algumas vezes na web burguesa e todas elas ou estão sem respostas, ou as respostas não são relevantes.





O que temos e o que queremos obter

Nós temos:





  • Certificado SSL / TLS comercial curinga;





  • Um servidor de diretório que pode fazer LDAP (no meu caso, Apache DS);





  • O servidor NiFi no qual o desenvolvimento está ocorrendo;





  • Servidor NiFi em um ambiente de mercearia;





  • Servidor de registro NiFi.





Nós queremos:





  • Autenticação, autorização e HTTPS. Além disso, o NiFi não permite que você habilite mecanismos de autenticação integrados sem HTTPS;





  • Gerenciamento centralizado de usuários em um servidor de diretório;





  • Fluxo de controle de versão e grupo de processos de maneira semelhante ao gith;





  • .





:

  1. Apache DS;





  2. Java Keystore Java Truststore, HTTPS;





  3. LDAP NiFi;





  4. LDAP NiFi Registry;





  5. ;





  6. NiFi NiFi Registry;





  7. NiFi NiFi Cli.





wildcard : *.love.you;





FQDN Apache DS: auth.love.you;





Apache DS: uid=admin,ou=system; : secret;





Apache DS love.you: dc=love,dc=you;





Apache DS nifiAdmin nifiUser NiFi NiFi Registry;





Apache DS nifiAdmins nifiUsers NiFi NiFi Registry;





FQDN NiFi, : nifi-deve.love.you;





FQDN NiFi Registry: nifi-re.love.you;





FQDN () NiFi: nifi.love.you;





, NiFi: $NIFI_HOME;





, NiFi Registry: $NIFI_REGISTRY_HOME;





, NiFi Toolkit: $NIFI_TOOLKIT_HOME;





, : /love/conf/cert;





PKCS12, Java Keystore Java Truststore:

sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO







Apache DS

, , Apache DS . . Azure AD, " ", NiFi, AD' . Apache DS , , Apache.





. , , , , . - .





10389 . Apache Directory Studio. , , .





New connection:





:





: uid=admin,ou=system secret. Check Authentication:





.





. - member objectclass inetOrgPerson. LDAP Browser' ou=schema Reload Entry:





cn=inetorgperson ou=objectClasses. m-oid=2.16.840.1.113730.3.2.2. Open With -> LDIF Entry Editor:





m-may: member Ctrl+S:





Open Configuration:





Advanced Partition configuration...





Add, ID Suffix. Ctrl+S:





: . . . :





. - LDIF . - import.ldif .





import.ldif 
dn: ou=people,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: people

dn: ou=groups,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: groups

dn: cn=nifiUsers,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiUsers
uniqueMember: cn=nifiUser,ou=people,dc=love,dc=you

dn: cn=nifiAdmins,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiAdmins
uniqueMember: cn=nifiAdmin,ou=people,dc=love,dc=you

dn: cn=nifiUser,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiUser
description: A nifiUser user
sn: nifiUser
uid: nifiUser
mail: nifiUser@love.you
userpassword: password
member: cn=nifiUsers,ou=groups,dc=love,dc=you

dn: cn=nifiAdmin,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiAdmin
description: A nifiAdmin user
sn: nifiAdmin
uid: nifiAdmin
mail: nifiAdmin@love.you
userpassword: password
member: cn=nifiAdmins,ou=groups,dc=love,dc=you

dn: cn=*.love.you,ou=people,dc=love,dc=you
objectClass: person
objectClass: top
cn: *.love.you
sn: *.love.you

LDIF Import:





:





:





cn=*.love.you

, ou=people cn=*.love.you, objectClass: person:





, : , *.love.you. NiFi Registry.





Java Keystore Java Truststore

, : PKCS12 JKS. , NiFi .





:





  • PEM-: fullchain.crt;





  • PEM-: private.key;





  • certificate authority (CA) PEM-: cacert.pem;





, . , .

. :





-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----

.





, .





, , , ( ) . CA .





Java Keystore . PKCS12:





openssl pkcs12 -export -out nifi.p12 -inkey private.key -in fullchain.crt -name nifi-key

PKCS12 . .





Java Keystore PKCS12 :





keytool -importkeystore -srckeystore nifi.p12 -srcstoretype pkcs12 -srcalias nifi-key -destkeystore keystore.jks -deststoretype jks -destalias nifi-key

, . Java Keystore, .





Java Truststore:





keytool -importcert -alias nifi-cert -file cacert.pem -keystore truststore.jks

Java Truststore. . Keystore Truststore. . .





JKS NiFi, , NiFi NiFi Registry, /love/conf/cert/;





NiFi , , $NIFI_HOME/config/nifi.properties, .





nifi.properties 
# Site to Site properties
nifi.remote.input.host=nifi-deve.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443

# web properties #
nifi.web.http.host=
nifi.web.http.port=

nifi.web.https.host=nifi-deve.love.you
nifi.web.https.port=9443

# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider

: nifi.web.http.host nifi.web.http.port. , , NiFi .

21 - , .





sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO



, .





NiFi , $NIFI_HOME/config/nifi.properties, .





nifi.properties 
# Site to Site properties
nifi.remote.input.host=nifi.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443

# web properties #
nifi.web.http.host=
nifi.web.http.port=

nifi.web.https.host=nifi.love.you
nifi.web.https.port=9443

# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider

# Identity Mapping Properties #
nifi.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE

: . .

21 - , .





sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO



, .





NiFi Registry, $NIFI_REGISTRY_HOME/config/nifi-registry.properties, .





nifi-registry.properties 
# web properties #
nifi.registry.web.http.host=
nifi.registry.web.http.port=

nifi.registry.web.https.host=nifi-re.love.you
nifi.registry.web.https.port=8443

# security properties #
nifi.registry.security.keystore=/abc/conf/cert/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.truststore=/abc/conf/cert/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.needClientAuth=false
nifi.registry.security.identity.provider=ldap-identity-provider

# Identity Mapping Properties #
nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE

: . .

16 - , .





sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO



, .





LDAP NiFi

LDAP : $NIFI_HOME/conf/login-identity-providers.xml $NIFI_HOME/conf/authorizers.xml





login-identity-providers.xml . LDAP- . .





login-identity-providers.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<loginIdentityProviders>
    <provider>
        <identifier>ldap-provider</identifier>
        <class>org.apache.nifi.ldap.LdapProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>

        <property name="Manager DN">uid=admin,ou=system</property>
        <property name="Manager Password">secret</property>

        <property name="TLS - Keystore"></property>
        <property name="TLS - Keystore Password"></property>
        <property name="TLS - Keystore Type"></property>
        <property name="TLS - Truststore"></property>
        <property name="TLS - Truststore Password"></property>
        <property name="TLS - Truststore Type"></property>
        <property name="TLS - Client Auth"></property>
        <property name="TLS - Protocol"></property>
        <property name="TLS - Shutdown Gracefully"></property>

        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>

        <property name="Url">ldap://auth.love.you:10389</property>
        <property name="User Search Base">ou=people,dc=love,dc=you</property>
        <property name="User Search Filter">(uid={0})</property>

        <property name="Identity Strategy">USE_USERNAME</property>
        <property name="Authentication Expiration">12 hours</property>
    </provider>
</loginIdentityProviders>

, 27 29. 27 , , LDAP-. {0} - , User NiFi; uid - , , . , nifiUser.





, . ApacheDS.





29 - : (USE_USERNAME), DN (USE_DN). , , . DN .





authorizers.xml , NiFi "" . .





authorizers.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
    <userGroupProvider>
        <identifier>ldap-user-group-provider</identifier>
        <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>

        <property name="Manager DN">uid=admin,ou=system</property>
        <property name="Manager Password">secret</property>

        <property name="TLS - Keystore"></property>
        <property name="TLS - Keystore Password"></property>
        <property name="TLS - Keystore Type"></property>
        <property name="TLS - Truststore"></property>
        <property name="TLS - Truststore Password"></property>
        <property name="TLS - Truststore Type"></property>
        <property name="TLS - Client Auth"></property>
        <property name="TLS - Protocol"></property>
        <property name="TLS - Shutdown Gracefully"></property>

        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>

        <property name="Url">ldap://auth.love.you:10389</property>
        <property name="Page Size"></property>
        <property name="Sync Interval">1 mins</property>
        <property name="Group Membership - Enforce Case Sensitivity">false</property>

        <property name="User Search Base">ou=people,dc=love,dc=you</property>
        <property name="User Object Class">person</property>
        <property name="User Search Scope">ONE_LEVEL</property>
        <property name="User Search Filter"></property>
        <property name="User Identity Attribute">cn</property>
        <property name="User Group Name Attribute">member</property>
        <property name="User Group Name Attribute - Referenced Group Attribute"></property>

        <property name="Group Search Base">ou=groups,dc=love,dc=you</property>
        <property name="Group Object Class">groupOfUniqueNames</property>
        <property name="Group Search Scope">ONE_LEVEL</property>
        <property name="Group Search Filter"></property>
        <property name="Group Name Attribute">cn</property>
        <property name="Group Member Attribute"></property>
        <property name="Group Member Attribute - Referenced User Attribute"></property>
    </userGroupProvider>

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">ldap-user-group-provider</property>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">nifiAdmin</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Node Identity 1"></property>
        <property name="Node Group"></property>
    </accessPolicyProvider>

    <authorizer>
        <identifier>managed-authorizer</identifier>
        <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
        <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>
</authorizers>

:

31 - objectClass ApacheDS, ;

35 - , ;

39 - objectClass, ;

50 - , ( identifier: <identifier>ldap-user-group-provider</identifier> <userGroupProvider>);

52 - , - "" .





NiFi nifiAdmin , ldif-.





LDAP NiFi Registry

NiFi Registry NiFi. $NIFI_REGISTRY_HOME/conf/identity-providers.xml .





identity-providers.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<identityProviders>
    <provider>
        <identifier>ldap-identity-provider</identifier>
        <class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>

        <property name="Manager DN">uid=admin,ou=system</property>
        <property name="Manager Password">secret</property>

        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>

        <property name="Url">ldap://auth.love.you:10389</property>
        <property name="User Search Base">ou=people,dc=love,dc=you</property>
        <property name="User Search Filter">(uid={0})</property>

        <property name="Identity Strategy">USE_USERNAME</property>
        <property name="Authentication Expiration">12 hours</property>
    </provider>
</identityProviders>

$NIFI_REGISTRY_HOME/conf/authorizers.xml .





authorizers.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
    <userGroupProvider>
        <identifier>ldap-user-group-provider</identifier>
        <class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>

        <property name="Manager DN">uid=admin,ou=system</property>
        <property name="Manager Password">secret</property>

        <property name="TLS - Keystore"></property>
        <property name="TLS - Keystore Password"></property>
        <property name="TLS - Keystore Type"></property>
        <property name="TLS - Truststore"></property>
        <property name="TLS - Truststore Password"></property>
        <property name="TLS - Truststore Type"></property>
        <property name="TLS - Client Auth"></property>
        <property name="TLS - Protocol"></property>
        <property name="TLS - Shutdown Gracefully"></property>

        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>

        <property name="Url">ldap://auth.love.you:10389</property>
        <property name="Page Size"></property>
        <property name="Sync Interval">1 mins</property>
        <property name="Group Membership - Enforce Case Sensitivity">false</property>

        <property name="User Search Base">ou=people,dc=love,dc=you</property>
        <property name="User Object Class">person</property>
        <property name="User Search Scope">ONE_LEVEL</property>
        <property name="User Search Filter"></property>
        <property name="User Identity Attribute">cn</property>
        <property name="User Group Name Attribute">member</property>
        <property name="User Group Name Attribute - Referenced Group Attribute"></property>

        <property name="Group Search Base">ou=groups,dc=love,dc=you</property>
        <property name="Group Object Class">groupOfUniqueNames</property>
        <property name="Group Search Scope">ONE_LEVEL</property>
        <property name="Group Search Filter"></property>
        <property name="Group Name Attribute">cn</property>
        <property name="Group Member Attribute"></property>
        <property name="Group Member Attribute - Referenced User Attribute"></property>
    </userGroupProvider>

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
        <property name="User Group Provider">ldap-user-group-provider</property>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">nifiAdmin</property>
        <property name="NiFi Group Name"></property>
    </accessPolicyProvider>

    <authorizer>
        <identifier>managed-authorizer</identifier>
        <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
        <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>

</authorizers>

NiFi Registry nifiAdmin , ldif-.





Identity Mapping Properties nifi.properties nifi-registry.properties.





, NiFi NiFi Registry.





, "Proxy user requests". , , . , " " " ", , . , , . , . . .





? , NiFi NiFi Registry , . NiFi Registry NiFi , NiFi Java Keystore , , NiFi Registry. , wildcard , CN=*.love.you.





!

, NiFi . , , CN . . , CN cn. NiFi .





. ApacheDS cn . . . : . NiFi , : nifiUser, nifiAdmin *.love.you. cn=....





, , , , .





, , ....mapping.pattern.dn capture group, ....mapping.value.dn . CN=*.love.you *.love.you, ApacheDS.





NiFi NiFi Registry

NiFi . , , NiFi . , .





Operate:





Policies , :





, *.love.you proxy user requests:





Controller Settings:





REGISTRY CLIENTS:





, + , , :





NiFi Registry, , :





USERS :





:





Bucket Test





NiFi. process group Test. process group Version -> Start version control:





:





SAVE process group :





process group .





NiFi NiFi Cli

, process group flow Registry, () NiFi.





NiFi Toolkit NiFi .





: $NIFI_TOOLKIT_HOME/config/reg.properties $NIFI_TOOLKIT_HOME/config/nifi-prod.properties:





NiFi Registry:





reg.properties 
baseUrl=https://nifi-re.love.you:8443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin

sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO



, .





() NiFi:





nifi-prod.properties 
baseUrl=https://nifi.love.you:9443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin

sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO



, .





:





  1. NiFi Registry ID , flow.





  2. flow ID ID flow, () NiFi.





  3. flow;





  4. flow () NiFi.





$NIFI_TOOLKIT_HOME/bin :





:





./cli.sh registry list-buckets -p ../conf/reg.properties

:





#   Name   Id                                     Description
-   ----   ------------------------------------   -----------
1   Test   84744b22-0e2e-4342-b6ab-6658ff0e54ec   (empty)

flow 84744b22-0e2e-4342-b6ab-6658ff0e54ec:





./cli.sh registry list-flows -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -p ../conf/reg.properties

:





#   Name                 Id                                     Description
-   ------------------   ------------------------------------   ------------------
1   PG versioning test   bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6   PG versioning test

flow:





./cli.sh registry list-flow-versions -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -p ../conf/reg.properties

:





Ver   Date                         Author      Message
---   --------------------------   ---------   ------------------
1     Thu, Apr 22 2021 09:28 UTC   nifiAdmin   PG versioning test

flow () NiFi:





./cli.sh nifi pg-import -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -fv 1 -p ../conf/nifi-prod.properties

:





f90df705-0178-1000-04fe-45ba27e76f99

- ID process group () NiFi:





, HowTo . , , (, NiFi Registry), "TRACE" NiFi NiFi Registry . , , .





No que diz respeito a CI / CD, não desenhei toda a cadeia. Todos têm seus próprios padrões e cenários, e o pipeline em si seria off -topic. E o exemplo acima NiFi Cli, eu acho, é um bom ponto de partida para implementar seu próprio esquema de entrega.





Obrigado a todos que leram e espero que o texto tenha sido útil.







More articles:


All Articles