ENT-6193, CFE-3421: Added policy function classfilterdata()

[larsewi]

• evalfunction.c: Removed trailing whitespace
• Added policy function classfilterdata()
• Added acceptance tests for classfilterdata()
• Collecting classfilterdata policy function
• Added examples for classfilterdata()

[olehermanse]

@cf-bottom jenkins, please

[cf-bottom]

Edit: This build is green, but parts of it was not run (DTs).

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12330/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12330/

[olehermanse]

@cf-bottom please build in Jenkins again :)

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12354/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12354/

[olehermanse]

@cf-bottom jenkins, please

[cf-bottom]

Lars had already done a rebuild so I cancelled tom's build and s/12359/12358/ here.

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12358/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12358/

[olehermanse]

@cf-bottom let's test this in Jenkins, please

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12360/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12360/

[larsewi]

^ Started it again without Debian builds (they are not working ATM)

[larsewi]

@cf-bottom let's test this in Jenkins, please

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12366/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12366/

[craigcomstock]

@cf-bottom jenkins please after fixing up the build host setup some more...

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12370/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12370/

[nickanderson]

I hit a crash when the first argument is a function, e.g. readjson() to pull the json from a file on disk instead of explicitly loading it separately first. Referencing a data container does work for me.

https://northerntech.atlassian.net/browse/CFE-3421?focusedCommentId=123161

[larsewi]

I hit a crash when the first argument is a function, e.g. readjson() to pull the json from a file on disk instead of explicitly loading it separately first. Referencing a data container does work for me.

Good find @nickanderson. I fixed the segmentation fault. The issue was that I had wrongfully set the function to have a variable amount of arguments (through copy pasta).

The first argument however, requires a CFEngine variable identifier or inline JSON. This is similar to other functions (like storejson(), dataexpand(), and many more. If you want to be able to nest function calls into these functions as well, we should create a separate ticket.

[larsewi]

@cf-bottom Jenkins please :)

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12373/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12373/

[nickanderson]

@larsewi

OK, I tested the latest PR build and see it's no longer dumping core and now it errors fatally:

$ cat classfilterdata.cf 
#!/var/cfengine/bin/cf-agent -f-
body file control
{
      inputs => { '$(sys.libdir)/stdlib.cf' };
}

bundle agent __main__
{
  classes:
      "role_1";

  vars:
      "filtered"
        data => classfilterdata( readjson( "/tmp/array_of_objects.json" ),
                                           "array_of_objects",
                                           "ifvarclass" );
}
nickanderson@precision-5570:/tmp$ cat array_of_objects.json 
[
  {
    "file": "/etc/grafana/backup_grafana_db.sh",
    "from": "$(sys.workdir)/lfk/etc/grafana/backup_grafana_db.sh",
    "upstream_url": "https://bitb0.lfk.acme.com/rest/api/1.0/projects/BATCH/repos/b2cfe/raw/etc/grafana/backup_grafana_db.sh?at=refs%2Fheads%2Fmain",
    "upstream_headers": [
      "Authorization: Bearer $(pmp_password[b2cfe_reader])"
    ],
    "write_method": "http",
    "mode": "755",
    "owner": "root",
    "group": "root",
    "ifvarclass": "role_1"
  },
  {
    "file": "/etc/grafana/grafana.ini",
    "from": "$(sys.workdir)/lfk/etc/grafana/grafana.ini",
    "upstream_url": "https://bitb0.lfk.acme.com/rest/api/1.0/projects/BATCH/repos/b2cfe/raw/etc/grafana/grafana.ini?at=refs%2Fheads%2Fmain",
    "upstream_headers": [
      "Authorization: Bearer $(pmp_password[b2cfe_reader])"
    ],
    "write_method": "http",
    "mode": "644",
    "owner": "root",
    "group": "root",
    "ifvarclass": "role_2"
  },
  {
    "file": "/etc/pki/tls/certs/AAACertificateServices.cer",
    "from": "$(sys.workdir)/lfk/AAACertificateServices.cer",
    "upstream_url": "https://bitb0.lfk.acme.com/rest/api/1.0/projects/itpub/repos/sslcerts/raw/AAACertificateServices.cer?at=refs%2Fheads%2Fmain",
    "upstream_headers": [
      "Authorization: Bearer $(pmp_password[itpub_sslcerts])"
    ],
    "write_method": "http",
    "mode": "644",
    "owner": "root",
    "group": "root",
      "ifvarclass": "(role_1|role_2)"
  }
]
nickanderson@precision-5570:/tmp$ cf-agent -KIf ./classfilterdata.cf 
   error: Argument template mismatch handling function classfilterdata( {"array_of_objects","ifvarclass"})
   error:   arg[0] range .*      array_of_objects 
   error:   arg[1] range array_of_arrays,array_of_objects,auto   array_of_objects 
   error:   arg[2] range .*      array_of_objects 
   error: Fatal CFEngine error: Bad arguments
   error: Policy failed validation with command '"/home/nickanderson/.cfagent/bin/cf-promises" -c "./classfilterdata.cf"'
   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.
   error: Error reading CFEngine policy. Exiting...
nickanderson@precision-5570:/tmp$ cf-agent -V
CFEngine Core 3.27.0a.16a7b10de
CFEngine Enterprise 3.27.0a.fd2ed860d

But, I think this function should have "Collecting" (https://docs.cfengine.com/docs/3.24/reference-functions.html#collecting-functions) behavior like data_expand() and storejson(). Function arguments marked as collecting can take their value from various sources and the data is normalized to JSON internally so that there is consistent behavior.

This is what allows a function to take a data container specified by it's bare name (mycontainer, bundle.mycontainer, or namespace:bundle.mycontainer), or with the @ syntax ( @(mycontainer), @(bundle.mycontainer), @(namespace:bundle.mycontainer) or a function that returns a data container.

I made this example that illustrates the behavior for data_expand() and storejson():

bundle agent __main__
{
  vars:
      "filename" string => "$(this.promise_dirname)/data.json";

  methods:
      "init";
      "test";
}

bundle agent init
{
  files:
      "$(main.filename)"
        content => "{ \"CFEngine version\": \"$(const.dollar)(sys.cf_version)\"}";

}

bundle agent test
{
  vars:

      # Here, per the documentation data_expand() takes a data container but you
      # can see from the example that it can take a function that returns a data
      # container.

      "expanded"
        data => data_expand( readjson( $(main.filename)  )),
        if => fileexists( $(main.filename) );

  reports:
      # Here storejson() takes the unqualified name of a data container
      "storejson( expanded ): $(with)"
        with => storejson( expanded );

      # Here storejson() takes readjson() as it's argument

      "storejson( readjson( \"$(main.filename)\" ) ): $(with)"
        with => storejson( readjson( "$(main.filename)" ) );
}

$ cf-agent -Kf ./test.cf 
R: storejson( expanded ): {
  "CFEngine version": "3.27.0a.3cf9d3922"
}
R: storejson( readjson( "/tmp/./data.json" ) ): {
  "CFEngine version": "$(sys.cf_version)"
}

Does that make sense?

[nickanderson]

I think it should have collecting behavior. https://docs.cfengine.com/docs/3.24/reference-functions.html#collecting-functions

[larsewi]

Thanks @nickanderson, I was not aware of this. I marked the function arguments as collecting and created an acceptance test to prove that it can now take nested function calls as arguments.

[larsewi]

@cf-bottom Jenkins please :)

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/12376/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-12376/

Added collecting behavior as requested