Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fedora iot-simplified-installer fdo re-encryption failed #3726

Closed
yih-redhat opened this issue Oct 9, 2023 · 11 comments
Closed

fedora iot-simplified-installer fdo re-encryption failed #3726

yih-redhat opened this issue Oct 9, 2023 · 11 comments

Comments

@yih-redhat
Copy link
Contributor

yih-redhat commented Oct 9, 2023 •
edited
Loading

Describe the bug
provision edge vm with iot-simplified-installer, install a failing health check unit and rollback, then check fdo re-encryption by command "cryptsetup luksDump /dev/vda3". The expected result is there is no "cipher_null-ecb" in output, but actually "cipher_null-ecb" is in the output.
The same test passed on rhel and centos-stream, the difference is we check /dev/vda4 on these os, with command "cryptsetup luksDump /dev/vda4", so I guess maybe the root reason is that fedora image only has /dev/vda3, and rhel/centos has /dev/vda4.

Environment

  • OS version (/etc/os-release and /etc/redhat-release):
  • osbuild-composer version (rpm -qi osbuild-composer)

To Reproduce
Steps to reproduce the behavior:

  1. build iot-simplified-installer and provision vm.
  2. install sanely failing health check unit and reboot
    "rpm-ostree install --cache-only https://s3.amazonaws.com/org.osbuild.test-dependencies/greenboot-failing-unit-1.0-1.el8.noarch.rpm --reboot"
  3. vm failed to boot and then rollback automatically.
  4. check fdo re-encryption status.
    [simple@localhost ~]$ sudo cryptsetup luksDump /dev/vda3
    [sudo] password for simple:
    LUKS header information
    Version: 2
    Epoch: 6
    Metadata area: 16384 [bytes]
    Keyslots area: 16744448 [bytes]
    UUID: 05ad1795-54bc-4a57-bb01-9082f86a774d
    Label: crypt_root
    Subsystem: (no subsystem)
    Flags: (no flags)

Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: cipher_null-ecb
sector: 512 [bytes]

Keyslots:
1: luks2
Key: 256 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: pbkdf2
Hash: sha256
Iterations: 1000
Salt: 6c 19 c0 8e 05 f0 05 21 42 70 98 5a 07 c9 19 8a
d0 7a d8 ef 16 14 95 be 94 9e d2 d8 46 bf 16 0f
AF stripes: 4000
AF hash: sha256
Area offset:163840 [bytes]
Area length:131072 [bytes]
Digest ID: 0
Tokens:
0: clevis
Keyslot: 1
Digests:
0: pbkdf2
Hash: sha256
Iterations: 1000
Salt: 75 96 82 66 56 55 02 a1 0a 63 58 db b2 c9 60 fd
3b cd 8d fe ef cf 39 76 73 7d 68 8e b0 6f f7 aa
Digest: b9 fb 7b a8 6f 2b 91 20 e2 8f b7 b4 2a 6f 67 09
7e bf b3 2b 45 2b c1 1c be 23 d9 dc e0 54 f2 48

Expected behavior
there is no "cipher_null-ecb" in output

Additional context
Add any other context about the problem here.
@yih-redhat
Copy link
Contributor Author

yih-redhat commented Oct 9, 2023 •
edited
Loading

@runcom @7flying Could you please take a look of this bug? And I found a similar bug was filed before https://bugzilla.redhat.com/show_bug.cgi?id=2220851

@yih-redhat yih-redhat mentioned this issue Oct 9, 2023
Closed
3 tasks
@yih-redhat
Copy link
Contributor Author

yih-redhat commented Oct 9, 2023 •
edited
Loading

fdo client log:

[simple@localhost home]$ journalctl -u fdo-client-linuxapp.service
Display all 177 possibilities? (y or n)
[simple@localhost home]$ journalctl -u fdo-client-linuxapp.service
Oct 09 13:49:10 localhost.localdomain systemd[1]: Starting fdo-client-linuxapp.service - FDO client...
Oct 09 13:49:10 localhost.localdomain fdo-client-linuxapp[1077]: 2023-10-09T13:49:10.386Z INFO fdo_client_linuxapp > No usable device>
Oct 09 13:49:10 localhost.localdomain systemd[1]: fdo-client-linuxapp.service: Deactivated successfully.
Oct 09 13:49:10 localhost.localdomain systemd[1]: Finished fdo-client-linuxapp.service - FDO client.
-- Boot cd6c4a0071004e97bf4bae580ac5d3d1 --
Oct 09 13:50:43 localhost.localdomain systemd[1]: Starting fdo-client-linuxapp.service - FDO client...
Oct 09 13:50:43 localhost.localdomain fdo-client-linuxapp[956]: 2023-10-09T13:50:43.400Z INFO fdo_client_linuxapp > No usable device >
Oct 09 13:50:43 localhost.localdomain mv[976]: /usr/bin/mv: cannot stat '/boot/device-credentials': No such file or directory
Oct 09 13:50:43 localhost.localdomain systemd[1]: fdo-client-linuxapp.service: Deactivated successfully.
Oct 09 13:50:43 localhost.localdomain systemd[1]: Finished fdo-client-linuxapp.service - FDO client.

fdo aio serviceinfo config file:
[root@yih-f38 ~]# cat /etc/fdo/aio/configs/serviceinfo_api_server.yml
service_info:
initial_user:
username: fdouser
sshkeys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkMAjwQdrK0LBzcyQxEsJQIxlRNQaKbH/eGI3lPzoFumY+HdxoktJKPDZCibB2LyIfOHu5qFj4HSeLSFy18TzydrgQiZxIgrZOvQyfaEutHxfNLbDxU5GSFqss43Nv3qnCCvac8K62m/nWmhQvyuGXZuRs/VFA/RxzjH8TYYQXLv0c66LXX2cH6Xs92gkL1YKjW0EEkySb+MsMGSl8CzFLlFHPNNRO9neAreLHAVL0xzPsHFDYti4Hb2xnL7LEy7iTB5e0gmcZ1/jyfIOc70dkq1a+/AXxCSGsDcdWahwBoKOYDRG0JJeYG3cw2zJXYp2H4/H3wVE0wxirejwkIwY+2BGOcO7W3LUHuWouPQ4idv+gZznDIz+Vt14IJyiiVz5ADNM9zeqHAYwQHvoj1/xokiLRbsH+zC+QzHsXHx4g7LNfyDKsX7rKmV9AKRQi6z0ypwAozKF3eTFHaUJNnEv4F5DJ5qiZiNtMv1jyJ+p4c2QZwSPEeSivCkegkIIuBWU=
fedora@yih-f38
files:
- path: /etc/sudoers.d/fdouser
source_path: /var/lib/fdo/fdouser
commands: null
diskencryption_clevis:
- disk_label: /dev/vda3
reencrypt: true
binding:
pin: tpm2
config: '{}'
additional_serviceinfo: null
after_onboarding_reboot: false
bind: 0.0.0.0:8083
service_info_auth_token: c3iJ+YAIT7PzmCA90z9bLISn2/OdJwrXVm9e4C258ig=
admin_auth_token: pfQHrBgMVvPwzctW439fsqtF4urNi/Mxc10pDcdp+Rc=
device_specific_store_driver:
Directory:
path: /etc/fdo/aio/stores/serviceinfo_api_devices

@7flying
Copy link
Member

7flying commented Oct 9, 2023

@yih-redhat we need the logs from the manufacturing-client.service, if you have them

@runcom @7flying Could you please take a look of this bug? And I found a similar bug was filed before https://bugzilla.redhat.com/show_bug.cgi?id=2220851

that one was a TPM issue, don't know yet if this will also be the case, but we'll need the manufacturing logs.

Thanks

@henrywang
Copy link
Member

henrywang commented Oct 9, 2023 •
edited
Loading

This might be selinux issue, I think. @yih-redhat Could you please check the selinux fix in RHEL 9 in Fedora? Thanks.

@yih-redhat
Copy link
Contributor Author

This might be selinux issue, I think. @yih-redhat Could you please check the selinux fix in RHEL 9 in Fedora? Thanks.

sure, could you please let me know the steps to check the selinux fix?

@yih-redhat
Copy link
Contributor Author

I do found below denied avc log for /tmp/fdouser, but after I changed to use /var/lib/fdo/fdouser, I didn't see it anymore.

type=AVC msg=audit(10/09/2023 08:35:03.580:9660) : avc: denied { open } for pid=232024 comm=fdo-serviceinfo path=/tmp/fdouser dev="tmpfs" ino=251 scontext=system_u:system_r:fdo_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=0

@yih-redhat yih-redhat changed the title fedora iot-simplified-installer fdo encryption failed fedora iot-simplified-installer fdo re-encryption failed Oct 10, 2023
@henrywang
Copy link
Member

It's selinux issue. The selinux fdo fix in RHEL should be landed in Fedora.

@7flying
Copy link
Member

7flying commented Oct 19, 2023 •
edited
Loading

This is what I've got:

ct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.232Z INFO  fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.122.180:8081", "http://fe80::97e2:1716:6aa8:88ba:8081"]
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.233Z INFO  fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.122.180:8081"
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.277Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda3, pin: tpm2, config: {}, reencrypt: true
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain audit[1392]: AVC avc:  denied  { search } for  pid=1392 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:  2023-10-19T14:54:53.413Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]: Caused by:
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     0: Error executing clevis
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     1: Error executing disk encryption for disk label /dev/vda3
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     2: Error rebinding clevis
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     3: Error binding clevis
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:     4: Failed to bind clevis: ExitStatus(unix_wait_status(256)), stdout: , stderr:
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        Error: Password generation failed - required entropy too low for settings
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        Unable to generate a new key
Oct 19 14:54:53 localhost.localdomain fdo-client-linuxapp[1133]:        Error adding new binding to /dev/vda3

So, it is selinux, but not our typical case!

@7flying
Copy link
Member

7flying commented Oct 19, 2023

Linked the issue to our issue tracker: fedora-iot/iot-distro#8

@yih-redhat
Copy link
Contributor Author

Verifying this bug with build https://koji.fedoraproject.org/koji/buildinfo?buildID=2320649

@yih-redhat
Copy link
Contributor Author

Verified this bug with build https://koji.fedoraproject.org/koji/buildinfo?buildID=2320649, fixed, the fdo re-encryption works as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@henrywang @7flying @yih-redhat