I've being running newer version of just-backup-btrfs, which was configured to remove snapshots in batches ~ at least 3x100 at once (this is what I typically have in 1.5-2 days).
Snapshots transferring become much faster, however when I delete 300 snapshots at once, well... you can imagine what happens, but I can afford this on desktop.
Seekwatcher fails to run on my system with following error:
~> sudo seekwatcher -t find.trace -o find.png -p 'find /backup_hdd > /dev/null' -d /dev/sda1
Traceback (most recent call last):
File "/usr/bin/seekwatcher", line 58, in <module>
from seekwatcher import rundata
File "numpy.pxd", line 43, in seekwatcher.rundata (seekwatcher/rundata.c:7885)
ValueError: numpy.dtype does not appear to be the correct type object
I have no idea what does it mean, but generally I think if seeking because of fragmentation is a real cause of performance degradation, then this is something that BTRFS can improve, since I still have 65% of free space on BTRFS partition that receives snapshots and fragmentation in this case seems weird.
P.S. I've unsubscribed from mailing list, cc me on answers, please.
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
Skype: nazar-pc
Diaspora: nazarpc@xxxxxxxxxxxxxxxxxxxxxxx
Tox: A9D95C9AA5F7A3ED75D83D0292E22ACE84BA40E912185939414475AF28FD2B2A5C8EF5261249
On 18.03.16 16:22, Nazar Mokrynskyi wrote:
>> But seriously, what are you doing that you can't lose more than 15
>> minutes of? Couldn't it be even 20 minutes, or a half-hour or an hour
>> with 15 minute snapshots only on the ssds (yes, I know the raid0 factor,
>> but the question still applies)?
> This is artificial psychological limit. Sometimes when you're actively coding, it is quite sad to loose even 5 minutes of work, since productivity is not constant. This is why 15 minutes was chosen as something that is not too critical. There is no other real reason behind this limit other than how I feel it.
>
>> And perhaps more importantly for your data, btrfs is still considered
>> "stabilizing, not fully stable and mature". Use without backups is
>> highly discouraged, but I'd suggest that btrfs in its current state might
>> not be what you're looking for if you can't deal with loss of more than
>> 15 minutes worth of changes anyway.
>>
>> Be that as it may...
>>
>> Btrfs is definitely not yet optimized. In many cases it reads or writes
>> only one device at a time, for instance, even in RaidN configuration.
>> And there are definitely snapshot scaling issues altho at your newer 500
>> snapshots total that shouldn't be /too/ bad.
> As an (relatively) early adopter I'm fine using experimental stuff with extra safeties like backups (hey, I've used it even without those while back:)). I fully acknowledge what is current state of BTRFS and want to help make it even better by stressing issues that me and other users encounter, searching for solutions, etc.
>
>> Dealing with reality, regardless of how or why, you currently have a
>> situation of intolerably slow receives that needs addressed. From a
>> practical perspective you said an ssd for backups is ridiculous and I
>> can't disagree, but there's another "throw hardware at it" solution that
>> might be a bit more reasonable...
>>
>> Spinning rust hard drives are cheap. What about getting another one, and
>> alternating your backup receives between them? That would halve the load
>> to one every thirty minutes, without changing your 15-minute snapshot and
>> backup policy at all. =:^)
>>
>> So that gives you two choices for halving the load to the spinning rust.
>> Either decide you really can live with half-hour loss of data, or throw
>> only a relatively small amount of money (well, as long as you have room
>> to plug in another sata device anyway, otherwise...) at it for a second
>> backup device, and alternate between them.
> Yes, I'm leaning toward earning new hardware right now, fortunately, laptop allows me to insert 2 x mSATA + 2 x 2.5 SATA drives, so I have exactly 2.5 SATA slot free.
>
>> OTOH, since you mentioned possible coding, optimization might not be a
>> bad thing, if you're willing to put in the time necessary to get up to
>> speed with the code and can work with the other devs in terms of timing,
>> etc. But that will definitely take significant time even if you do it,
>> and the alternating backup solution can be put to use as soon as you can
>> get another device plugged in and setup. =:^)
> I'm not coding C/C++, so my capabilities to improve BTRFS itself are limited, but I'm always trying to find the reason and fix it instead of living with workarounds forever.
>
> I'll play with Seekwatcher and optimizing snapshots deletion and will post an update afterwards.
>
> Sincerely, Nazar Mokrynskyi
> github.com/nazar-pc
> Skype: nazar-pc
> Diaspora: nazarpc@xxxxxxxxxxxxxxxxxxxxxxx
> Tox: A9D95C9AA5F7A3ED75D83D0292E22ACE84BA40E912185939414475AF28FD2B2A5C8EF5261249
>
Attachment:
smime.p7s
Description: Кріптографічний підпис S/MIME
