26.04.2019 21:12, MASAKI Haruka пишет:
>> You have two subvolumes with the same received_uuid. Is the content of
>> these subvolumes the same? Is "missing" file present in both subvolumes?
>
> No, each subvolume name (190312-010100, 190315-001000) is timestamp.
> These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.
>
Which is your problem. Two subvolumes with the same received_uuid are
expected to have exactly the same content (received_uuid can be viewed
as dataset UUID). You send incremental stream against base uuid
9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive
looks for matching base subvolume and happens to pick 190312-010100.
There were patches to automatically remove received_uuid when subvolume
is changed to read-write. I do not know whether they were applied.
btrfs send/receive expects subvolumes to remain read-only. As soon as
you start fiddling with read-only status you already lost. btrfs should
have never allowed to change read-only into read-write in the first place.
> 190315-001000 has lost file,
> % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
> Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>
> but 190312-010100 doesn't.
> % ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
> Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>
>
>
>
> On Fri, 26 Apr 2019 20:30:07 +0300
> Andrei Borzenkov <arvidjaar@xxxxxxxxx> wrote:
>
>> 26.04.2019 7:50, MASAKI Haruka пишет:
>>> Thank you for your reply.
>>>
>>> # What does
>>>
>>> @Host A ("lily" master, new volume)
>>> aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv .
>>>
>>> @Host B ("daisy" backup, having volumes)
>>> aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500
>>>
>>> ( ...Transfared 11TB... )
>>>
>>> @Host A
>>> aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world
>>> Create a snapshot of '190315-001000' in './world'
>>> aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1
>>> Create a readonly snapshot of 'world' in './back1'
>>>
>>> @Host A
>>> aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555
>>> At subvol back1
>>> BTRFS_IOC_SEND returned 0
>>> joining genl thread
>>>
>>> @Host B
>>> root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
>>> At snapshot back1
>>> receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658
>>> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658
>>>
>>> # Host A current
>>>
>>> aki@lily /mnt % ls
>>> 190315-001000 back1 back2 world
>>> aki@lily /mnt % sudo btrfs sub li -Rqu .
>>> ID 258 gen 2698 top level 5 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
>>> ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid - uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
>>> ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
>>> ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid - uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2
>>>
>>>
>>> # Host B current
>>>
>>> aki@daisy /HySlave % pwd
>>> /HySlave
>>> aki@daisy /HySlave % ls
>>> 190312-010100 190315-001000 back1 back2
>>> [aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
>>> ID 256 gen 36238 top level 5 parent_uuid - received_uuid - uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
>>> ID 259 gen 35116 top level 256 parent_uuid - received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
>>> ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000
>>
>> You have two subvolumes with the same received_uuid. Is the content of
>> these subvolumes the same? Is "missing" file present in both subvolumes?
>>
>>> ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1
>>> ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2
>>>
>>> # Files missing
>>>
>>> aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>>>
>>> aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>>>
>>> aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>>>
>>> ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>>>
>>> aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid SMEE el_louis firstkiss gakuensai infinity-kid kid-memories-off ojyo_tokkyu samusupi wizards_harmony yuukyuu-chara-seala
>>>
>>>
>>> SMEE is missing.
>>>
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, 25 Apr 2019 20:55:59 +0300
>>> Andrei Borzenkov <arvidjaar@xxxxxxxxx> wrote:
>>>
>>>> 25.04.2019 16:34, MASAKI Haruka пишет:
>>>> [...]
>>>>
>>>> Please show
>>>>
>>>> 1. Full protocol of commands you executed (commands and their output),
>>>> e.g. captured using script.
>>>> 2. Commands used to determine missing files on both source and receive
>>>> sides.
>>>> 3. Listing of subvolumes on both source and receive sides with
>>>>
>>>> btrfs sub li -Rqu /mount/point
>>>>
>>>> [...]
>>>>
>>
