authoravatarclover caruso<git@paperclover.net>2026-10-16 13:52:08-07:00
committeravatarclover caruso<git@paperclover.net>2026-10-19 17:26:14-07:00
logd2b738a4ff1796471a1f7581f203e652392fc309
tree4f49e7e6a4a645a74c64b567e4cf129848d60768
parent132fa1379f8ba53b504a5bdda15a9cc58d5f44cd
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU

fix(notebook): keep edits waiting while the cache cannot reopen

After a failed COMMIT, a cold reopen decides whether the commit landed. When that reopen fails for now (too many open files, a busy cache, memory pressure), the cache stays closed instead of failing the section: the edits wait in memory, the reopen is retried once a second however busy the section is, and the commit is decided once it succeeds, so its edits land exactly once. The closed cache stays owned. The warm connection, never used again, keeps SQLite's lock until the replica holds <cache>-lock and lists the cache file for the process in its stead, and every cache open fails busy while either is held; the check runs once the opener holds SQLite's lock, so no second owner can publish and acknowledge the undecided commit and have it queued again. A replica dropped meanwhile keeps retrying and keeps the cache owned. An edit a caller blocks on is refused at once, as under #136. The reread that refusal needs waits for the cache: page reads answer from the section as it was, later edits wait to apply to the reread section, and once the cache reopens only the closed commit is decided before the edits apply again, as #136 does, so an edit on a refused page is refused rather than queued. The session reports the section's pages as changed after such a reread. Meanwhile reads of the cache answer WouldBlock and the sync worker carries on; sync_status reports the edits queued and waiting and a failure instead of erring, and the desktop counts unsaved changes through it. Only a cache file that is gone, or no longer a database, is lost: every waiting edit is refused with the error and the section stays closed. A cold open that comes back read-only is retried. - Edits made while Snowbound cannot reopen its cache after a failed save now keep waiting and save once it can. Fixes #148 Assisted-by: claude-opus-5.5

9 files changed, 1301 insertions(+), 334 deletions(-)

crates/notebook/README.md+9-3
......@@ -53,11 +53,17 @@ connection holds exclusive ownership between transactions, and a second open fai
5353busy. No network wait occurs in a local edit. An edit a session submitted that the cache
5454cannot write for now (a full disk, an I/O error) stays applied in memory and is written
5555again, in order, each second; meanwhile `written` returns `WouldBlock`. A replica dropped
56meanwhile keeps its cache open and its edits retrying until they are written or the cache is
56meanwhile keeps its cache owned and its edits retrying until they are written or the cache is
5757deleted, and only then releases it; `notebook::unwritten()` counts the edits waiting in the
5858process, for a host to ask before it quits. After a commit fails, a fresh connection
59decides whether it landed. An edit the cache could never write is refused alone, and the
60section thread rereads the queue.
59decides whether it landed. Until the cache reopens (too many open files, a busy cache), the
60edits wait the same way, reads of the cache answer `WouldBlock`, and the cache stays owned:
61the replica locks `<cache>-lock` and lists the cache for the process, and every open fails
62busy while it does. A cache gone or no longer a database refuses every edit waiting, as an
63error, and stays closed. An edit a caller blocks on, or one the cache could never write, is
64refused at once, and the section thread rereads the queue; while the cache is closed, that
65reread waits for it, page reads answer from the section as it was, and later edits wait to
66apply to the reread section; the session then reports the section's pages as `Changed`.
6167
6268## Sessions
6369
crates/notebook/src/lib.rs+129-6
......@@ -281,8 +281,8 @@ impl Replica {
281281 })
282282 }
283283
284 fn lock(&self) -> Result<MutexGuard<'_, Connection>> {
285 lock(&self.section.connection)
284 fn lock(&self) -> Result<Open<'_>> {
285 lock(&self.section.cache)
286286 }
287287
288288 /// Hands `request` to the section thread.
......@@ -494,10 +494,41 @@ impl Drop for Replica {
494494 }
495495}
496496
497fn lock(connection: &Mutex<Connection>) -> Result<MutexGuard<'_, Connection>> {
498 connection
497/// The open cache's connection, for as long as its lock is held.
498struct Open<'a>(MutexGuard<'a, working::Cache>);
499
500impl std::ops::Deref for Open<'_> {
501 type Target = Connection;
502
503 fn deref(&self) -> &Connection {
504 match &*self.0 {
505 working::Cache::Open(connection) => connection,
506 _ => unreachable!("Only an open cache is lent"),
507 }
508 }
509}
510
511impl std::ops::DerefMut for Open<'_> {
512 fn deref_mut(&mut self) -> &mut Connection {
513 match &mut *self.0 {
514 working::Cache::Open(connection) => connection,
515 _ => unreachable!("Only an open cache is lent"),
516 }
517 }
518}
519
520/// The cache's connection; `WouldBlock` while a failed commit keeps the cache closed.
521fn lock(cache: &Mutex<working::Cache>) -> Result<Open<'_>> {
522 let cache = cache
499523 .lock()
500 .map_err(|_| io::Error::other("Cache owner panicked").into())
524 .map_err(|_| io::Error::other("Cache owner panicked"))?;
525 match &*cache {
526 working::Cache::Open(_) => Ok(Open(cache)),
527 working::Cache::Closed { .. } => {
528 Err(io::Error::new(io::ErrorKind::WouldBlock, "The cache reopens once it can").into())
529 }
530 working::Cache::Lost(error) => Err(io::Error::other(error.clone()).into()),
531 }
501532}
502533
503534/// Tells the worker a burst of local edits is durable.
......@@ -558,9 +589,101 @@ fn signed(value: u64) -> Result<i64> {
558589 Ok(i64::try_from(value).map_err(|_| io::Error::from(io::ErrorKind::InvalidInput))?)
559590}
560591
592/// `connect`, busy while anyone owns the cache: SQLite's lock, or the `Owned` a section holds
593/// while a failed commit keeps the cache closed, checked once SQLite's is held, as that
594/// section takes its own before letting SQLite's go.
595fn cache_connection(path: &Path) -> Result<Connection> {
596 let connection = connect(path)?;
597 connection.query_row("SELECT count(*) FROM sqlite_schema", [], |_| Ok(()))?;
598 let owned = identity(path)
599 .is_ok_and(|identity| OWNED.lock().is_ok_and(|owned| owned.contains(&identity)))
600 || std::fs::File::open(owner_lock(path)).is_ok_and(|lock| {
601 matches!(
602 lock.try_lock_shared(),
603 Err(std::fs::TryLockError::WouldBlock)
604 )
605 });
606 if owned {
607 return Err(rusqlite::Error::SqliteFailure(
608 rusqlite::ffi::Error::new(rusqlite::ffi::SQLITE_BUSY),
609 Some("A section holds the cache while it reopens".into()),
610 )
611 .into());
612 }
613 Ok(connection)
614}
615
616/// The file beside a cache whose lock keeps it owned while a failed commit keeps it closed.
617fn owner_lock(cache: &Path) -> PathBuf {
618 let mut lock = cache.as_os_str().to_owned();
619 lock.push("-lock");
620 lock.into()
621}
622
623/// Caches an `Owned` holds in this process, where a lock on a network filesystem (`flock` over
624/// NFS on Linux) need not exclude.
625static OWNED: Mutex<std::collections::BTreeSet<Identity>> =
626 Mutex::new(std::collections::BTreeSet::new());
627
628/// A cache file however its path is spelled: its device and inode, or elsewhere its
629/// canonical path.
630#[cfg(unix)]
631type Identity = (u64, u64);
632#[cfg(not(unix))]
633type Identity = PathBuf;
634
635fn identity(cache: &Path) -> io::Result<Identity> {
636 #[cfg(unix)]
637 {
638 use std::os::unix::fs::MetadataExt;
639 let metadata = std::fs::metadata(cache)?;
640 Ok((metadata.dev(), metadata.ino()))
641 }
642 #[cfg(not(unix))]
643 std::fs::canonicalize(cache)
644}
645
646/// Ownership of a closed cache, which every open honours until it drops: `<cache>-lock`
647/// locked, and the cache listed in `OWNED`.
648struct Owned {
649 cache: PathBuf,
650 identity: Identity,
651 _lock: std::fs::File,
652}
653
654impl Owned {
655 fn take(cache: &Path) -> Result<Self> {
656 let lock = std::fs::File::options()
657 .create(true)
658 .truncate(false)
659 .write(true)
660 .open(owner_lock(cache))?;
661 lock.try_lock().map_err(io::Error::from)?;
662 let identity = identity(cache)?;
663 #[cfg_attr(unix, allow(clippy::clone_on_copy))]
664 (OWNED.lock())
665 .map_err(|_| io::Error::other("Cache owners panicked"))?
666 .insert(identity.clone());
667 Ok(Self {
668 cache: cache.to_owned(),
669 identity,
670 _lock: lock,
671 })
672 }
673}
674
675impl Drop for Owned {
676 fn drop(&mut self) {
677 if let Ok(mut owned) = OWNED.lock() {
678 owned.remove(&self.identity);
679 }
680 let _ = std::fs::remove_file(owner_lock(&self.cache));
681 }
682}
683
561684/// Opens a cache without writing to it: exclusive locking, a full sync of every commit
562685/// (`F_FULLFSYNC` on macOS, the WAL's syncs included) and foreign keys, each queried back.
563fn cache_connection(path: &Path) -> Result<Connection> {
686fn connect(path: &Path) -> Result<Connection> {
564687 #[cfg(target_arch = "wasm32")]
565688 fs::install_sqlite();
566689 let connection = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE)?;
crates/notebook/src/location.rs+16-6
......@@ -191,13 +191,23 @@ fn adopt(from: &Path, to: &Path) -> io::Result<()> {
191191 Ok(())
192192}
193193
194/// Removes a replica folder that holds no replica.
194/// Removes a replica folder that holds no replica, with the owner lock a crash may have left
195/// beside a cache moved away since (`crate::Owned`).
195196fn tidy(folder: &Path) {
196 let empty = fs::read_dir(folder).is_ok_and(|mut entries| {
197 entries.all(|entry| entry.is_ok_and(|entry| entry.file_name() == LOCATION))
198 });
199 if empty {
200 let _ = fs::remove_file(folder.join(LOCATION));
197 let Ok(entries) = fs::read_dir(folder).and_then(|entries| {
198 entries
199 .map(|entry| Ok(entry?.file_name()))
200 .collect::<io::Result<Vec<_>>>()
201 }) else {
202 return;
203 };
204 let left = |name: &std::ffi::OsString| {
205 name == LOCATION || name.to_string_lossy().ends_with(".sqlite-lock")
206 };
207 if entries.iter().all(left) {
208 for name in entries {
209 let _ = fs::remove_file(folder.join(name));
210 }
201211 let _ = fs::remove_dir(folder);
202212 }
203213}
crates/notebook/src/session.rs+32-11
......@@ -2186,7 +2186,8 @@ pub struct SyncStatus {
21862186 pub synced: Option<u64>,
21872187 /// Why the last step failed, until one reaches the section file again.
21882188 pub error: Option<io::Error>,
2189 /// Edits the section file does not hold yet, uncertain attempts included.
2189 /// Edits the section file does not hold yet, uncertain attempts and edits waiting for the
2190 /// cache included; while a failed commit keeps the cache closed, those queued before it.
21902191 pub queued: u64,
21912192}
21922193
......@@ -2237,11 +2238,23 @@ pub(crate) fn sync_status(
22372238 synced: impl FnOnce() -> Option<u64>,
22382239 observed: &Mutex<Observed>,
22392240) -> Result<SyncStatus> {
2240 let queued: i64 = crate::lock(&replica.section.connection)?.query_row(
2241 "SELECT count(*) FROM edits",
2242 [],
2243 |row| row.get(0),
2244 )?;
2241 let waiting = replica
2242 .section
2243 .waiting
2244 .load(std::sync::atomic::Ordering::Relaxed) as u64;
2245 let (queued, failed) = match &*(replica.section.cache.lock())
2246 .map_err(|_| io::Error::other("Cache owner panicked"))?
2247 {
2248 crate::working::Cache::Open(connection) => {
2249 let queued: i64 =
2250 connection.query_row("SELECT count(*) FROM edits", [], |row| row.get(0))?;
2251 (crate::unsigned(queued)?, None)
2252 }
2253 crate::working::Cache::Closed { queued, .. } => {
2254 (*queued, Some("The cache reopens once it can".to_owned()))
2255 }
2256 crate::working::Cache::Lost(error) => (0, Some(error.clone())),
2257 };
22452258 let observed = observed
22462259 .lock()
22472260 .map_err(|_| io::Error::other("Synchronization observer panicked"))?;
......@@ -2249,11 +2262,11 @@ pub(crate) fn sync_status(
22492262 // Read after `observed`: the worker stores the time before it clears the error, so a
22502263 // cleared error never shows beside the time of the step before it.
22512264 synced: synced(),
2252 error: observed
2253 .as_ref()
2254 .and_then(Option::as_ref)
2255 .map(|error| io::Error::new(error.kind(), error.to_string())),
2256 queued: crate::unsigned(queued)?,
2265 error: failed.map(io::Error::other).or_else(|| {
2266 (observed.as_ref().and_then(Option::as_ref))
2267 .map(|error| io::Error::new(error.kind(), error.to_string()))
2268 }),
2269 queued: queued + waiting,
22572270 })
22582271}
22592272
......@@ -2475,6 +2488,14 @@ impl Section {
24752488 }
24762489 });
24772490 let observed: Arc<Mutex<Observed>> = Arc::new(Mutex::new(None));
2491 if let Ok(mut reread) = replica.section.reread.lock() {
2492 let (sender, notify) = (sender.clone(), Arc::clone(&notify));
2493 *reread = Some(Box::new(move |spaces| {
2494 if sender.send(Event::Changed(spaces)).is_ok() {
2495 notify();
2496 }
2497 }));
2498 }
24782499 let mut section = Self {
24792500 file,
24802501 replica,
crates/notebook/src/session/tests.rs+21
......@@ -854,3 +854,24 @@ fn edits_the_cache_refuses_wait_in_order_and_land_once() {
854854 assert_eq!(waited[0].page.color, Some(0x0065_4321));
855855 assert!(Replica::open(&cache).unwrap().pending().unwrap().is_empty());
856856}
857
858#[test]
859fn a_cache_owned_while_closed_opens_for_nobody_else() {
860 let temporary = tempfile::tempdir().unwrap();
861 let cache = temporary.path().join("owned.sqlite");
862 let source = onestore::create_section("Owned.one", "Original", "Fixture").unwrap();
863 drop(Replica::create(&cache, &source).unwrap());
864 let named = PathBuf::from(crate::connect(&cache).unwrap().path().unwrap());
865 let owned = crate::Owned::take(&named).unwrap();
866 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
867 assert!(crate::closed(&cache).is_err_and(|error| error.busy()));
868 drop(owned);
869 assert!(!crate::owner_lock(&named).exists());
870 drop(Replica::open(&cache).unwrap());
871 // Another process holds only the lock beside the cache.
872 let lock = std::fs::File::create(crate::owner_lock(&named)).unwrap();
873 lock.try_lock().unwrap();
874 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
875 drop(lock);
876 drop(Replica::open(&cache).unwrap());
877}
crates/notebook/src/working.rs+467-234
......@@ -15,7 +15,7 @@ use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params};
1515use std::{
1616 collections::{BTreeMap, BTreeSet, VecDeque},
1717 io,
18 path::PathBuf,
18 path::{Path, PathBuf},
1919 sync::{
2020 Arc, Mutex, Weak,
2121 atomic::{AtomicUsize, Ordering},
......@@ -25,6 +25,7 @@ use std::{
2525
2626pub(crate) type Reply<T> = Box<dyn FnOnce(Result<T>) + Send>;
2727pub(crate) type Notice = Box<dyn Fn(&str) + Send>;
28type Changed = Box<dyn Fn(Vec<ExGuid>) + Send>;
2829
2930pub(crate) enum Request {
3031 Apply(Accepted),
......@@ -87,6 +88,34 @@ pub(crate) struct Sealed {
8788 pub transaction: Option<Transaction>,
8889}
8990
91/// The cache's connection, or what a failed commit leaves: the cache closed until it reopens
92/// and reads cold whether the commit whose last edit is `last` landed, `ids` naming its edits
93/// still waiting and `queued` the edits queued before it; or lost where it never will reopen.
94pub(crate) enum Cache {
95 Open(Connection),
96 Closed {
97 last: u64,
98 ids: Vec<u64>,
99 queued: u64,
100 owner: Owner,
101 },
102 Lost(String),
103}
104
105/// What keeps everyone else out of a closed cache: the warm connection, unused, until
106/// `crate::Owned` can be taken in its stead.
107pub(crate) enum Owner {
108 Warm(#[allow(dead_code)] Connection),
109 Owned(#[allow(dead_code)] crate::Owned),
110}
111
112/// Whether a failed commit keeps the cache closed.
113fn closed(cache: &Mutex<Cache>) -> bool {
114 cache
115 .lock()
116 .is_ok_and(|cache| matches!(*cache, Cache::Closed { .. }))
117}
118
90119impl Request {
91120 fn fail(self, kind: io::ErrorKind, message: &str) {
92121 let error = || io::Error::new(kind, message.to_owned()).into();
......@@ -112,7 +141,10 @@ impl Request {
112141/// keeps answering page reads from the section as it was; the new thread then takes the
113142/// requests over, so reads never wait for a rebuild.
114143pub(crate) struct Thread {
115 pub(crate) connection: Mutex<Connection>,
144 pub(crate) cache: Mutex<Cache>,
145 /// Where the cache reopens after a failed commit; none in the browser, where reopening
146 /// through its file system is unverified.
147 path: Option<PathBuf>,
116148 /// The sync worker, which a durable burst of edits wakes.
117149 pub(crate) worker: Mutex<Weak<Signal>>,
118150 /// Taken when the replica drops, which ends every section thread.
......@@ -126,7 +158,9 @@ pub(crate) struct Thread {
126158 /// A password-protected section's key, under which its queue is sealed too.
127159 pub(crate) key: Option<Key>,
128160 /// The edits the cache last refused for now, as `crate::unwritten` counts them.
129 waiting: AtomicUsize,
161 pub(crate) waiting: AtomicUsize,
162 /// Told the pages a reread after a refusal may change, which their views reload.
163 pub(crate) reread: Mutex<Option<Changed>>,
130164 /// Last, so the cache closes before background synchronization takes it over.
131165 pub(crate) released: crate::Released,
132166}
......@@ -309,18 +343,20 @@ mod serving {
309343 }
310344}
311345
312/// The next request; a second passes without one while `retry`.
346/// How often the cache is tried again while edits wait for it or it is closed.
347const RETRY: std::time::Duration = std::time::Duration::from_secs(1);
348
349/// The next request, or a timeout once `retry` passes without one.
313350#[cfg(not(target_arch = "wasm32"))]
314351async fn receive(
315352 requests: &mpsc::Receiver<Request>,
316 retry: bool,
353 retry: Option<std::time::Duration>,
317354) -> std::result::Result<Request, mpsc::RecvTimeoutError> {
318 if retry {
319 requests.recv_timeout(std::time::Duration::from_secs(1))
320 } else {
321 requests
355 match retry {
356 Some(wait) => requests.recv_timeout(wait),
357 None => requests
322358 .recv()
323 .map_err(|_| mpsc::RecvTimeoutError::Disconnected)
359 .map_err(|_| mpsc::RecvTimeoutError::Disconnected),
324360 }
325361}
326362
......@@ -329,7 +365,7 @@ async fn receive(
329365#[cfg(target_arch = "wasm32")]
330366async fn receive(
331367 requests: &mpsc::Receiver<Request>,
332 _retry: bool,
368 _retry: Option<std::time::Duration>,
333369) -> std::result::Result<Request, mpsc::RecvTimeoutError> {
334370 std::future::poll_fn(|_| match requests.try_recv() {
335371 Ok(request) => std::task::Poll::Ready(Ok(request)),
......@@ -349,7 +385,10 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr
349385 #[cfg(target_arch = "wasm32")]
350386 let (durable_wake, woken) = crate::task::channel();
351387 let shared = Arc::new(Thread {
352 connection: Mutex::new(connection),
388 path: (connection.path())
389 .filter(|_| cfg!(not(target_arch = "wasm32")))
390 .map(PathBuf::from),
391 cache: Mutex::new(Cache::Open(connection)),
353392 worker: Mutex::new(Weak::new()),
354393 sender: Mutex::new(Some(sender)),
355394 #[cfg(target_arch = "wasm32")]
......@@ -360,6 +399,7 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr
360399 threads: Mutex::new(Vec::new()),
361400 key,
362401 waiting: AtomicUsize::new(0),
402 reread: Mutex::new(None),
363403 released: crate::Released::default(),
364404 });
365405 #[cfg(target_arch = "wasm32")]
......@@ -393,7 +433,7 @@ async fn run(
393433) {
394434 loop {
395435 let arena = Arena::default();
396 let working = match Working::open(&arena, &shared.connection, shared.key.as_ref()) {
436 let working = match Working::open(&arena, &shared.cache, shared.key.as_ref()) {
397437 Ok(working) => working,
398438 Err(error) => {
399439 if let Some(ready) = ready.take() {
......@@ -451,19 +491,17 @@ enum Job {
451491fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) {
452492 let answer: Box<dyn FnOnce() + Send> = match job {
453493 Job::Reopen { reply } => Box::new(move || reply(Ok(()))),
454 Job::Rebase { image, reply } => {
455 match rebase(&shared.connection, image, shared.key.as_ref()) {
456 Ok(changed) => Box::new(move || reply(Ok(changed))),
457 Err(error) => {
458 reply(Err(error));
459 let _ = signal.send(Request::Resume);
460 return;
461 }
494 Job::Rebase { image, reply } => match rebase(&shared.cache, image, shared.key.as_ref()) {
495 Ok(changed) => Box::new(move || reply(Ok(changed))),
496 Err(error) => {
497 reply(Err(error));
498 let _ = signal.send(Request::Resume);
499 return;
462500 }
463 }
501 },
464502 };
465503 let arena = Arena::default();
466 let opened = Working::open(&arena, &shared.connection, shared.key.as_ref());
504 let opened = Working::open(&arena, &shared.cache, shared.key.as_ref());
467505 let (to, from) = mpsc::sync_channel(1);
468506 if signal.send(Request::Handover(to)).is_err() {
469507 return;
......@@ -492,40 +530,65 @@ fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) {
492530fn inline(shared: &Thread, job: Job) -> bool {
493531 match job {
494532 Job::Reopen { reply } => reply(Ok(())),
495 Job::Rebase { image, reply } => {
496 match rebase(&shared.connection, image, shared.key.as_ref()) {
497 Ok(changed) => reply(Ok(changed)),
498 Err(error) => {
499 reply(Err(error));
500 return false;
501 }
533 Job::Rebase { image, reply } => match rebase(&shared.cache, image, shared.key.as_ref()) {
534 Ok(changed) => reply(Ok(changed)),
535 Err(error) => {
536 reply(Err(error));
537 return false;
502538 }
503 }
539 },
504540 }
505541 true
506542}
507543
508/// Writes again each second the edits a dropped replica left waiting, until none do or their
509/// cache is gone, keeping the cache open and `Released` from firing meanwhile.
544/// Writes again each second the edits a dropped replica left waiting, until none do and the
545/// cache is open, or the cache is gone; the cache stays owned and `Released` unfired meanwhile.
546/// With `reread`, the closed commit is settled first and the edits applied again to the queue
547/// as it then stands, refusing those that no longer apply, as the section thread would have.
510548#[cfg(not(target_arch = "wasm32"))]
511fn orphan(shared: &Arc<Thread>, unwritten: Vec<Accepted>) {
549fn orphan(shared: &Arc<Thread>, mut open: Option<i64>, mut unwritten: Vec<Accepted>, reread: bool) {
512550 let shared = Arc::clone(shared);
513 let path = lock(&shared.connection)
514 .ok()
515 .and_then(|connection| connection.path().map(PathBuf::from));
516551 let spawned = std::thread::Builder::new()
517552 .name("onestore-unwritten".into())
518553 .spawn(move || {
519 let arena = Arena::default();
520 let Ok(mut working) = Working::open(&arena, &shared.connection, shared.key.as_ref())
521 else {
522 return;
523 };
524 working.unwritten = unwritten;
525554 // A cache deleted, as with its notebook, takes no edits.
526 while !working.unwritten.is_empty() && path.as_ref().is_some_and(|path| path.exists()) {
527 std::thread::sleep(std::time::Duration::from_secs(1));
528 let _ = working.flush(&shared);
555 let kept = || shared.path.as_ref().is_some_and(|path| path.exists());
556 let second = || std::thread::sleep(std::time::Duration::from_secs(1));
557 while closed(&shared.cache) && kept() {
558 second();
559 let _ = write(&shared, &mut open, &mut unwritten, !reread);
560 }
561 // An edit that applies part way is refused, and the others apply again to a fresh
562 // reading of the queue.
563 if reread {
564 'reread: loop {
565 let arena = Arena::default();
566 let mut working =
567 match Working::open(&arena, &shared.cache, shared.key.as_ref()) {
568 Ok(working) => working,
569 Err(error) => {
570 for edit in unwritten.drain(..) {
571 (edit.reply)(Err(io::Error::other(error.to_string()).into()));
572 }
573 break;
574 }
575 };
576 let mut edits = std::mem::take(&mut unwritten).into_iter();
577 while let Some(edit) = edits.next() {
578 if working.accept(edit) {
579 unwritten = std::mem::take(&mut working.unwritten);
580 unwritten.extend(edits);
581 continue 'reread;
582 }
583 }
584 open = working.open;
585 unwritten = std::mem::take(&mut working.unwritten);
586 break;
587 }
588 }
589 while !unwritten.is_empty() && kept() {
590 second();
591 let _ = write(&shared, &mut open, &mut unwritten, true);
529592 }
530593 });
531594 if let Err(error) = spawned {
......@@ -556,7 +619,7 @@ struct Working<'a> {
556619}
557620
558621impl<'a> Working<'a> {
559 fn open(arena: &'a Arena, connection: &Mutex<Connection>, key: Option<&Key>) -> Result<Self> {
622 fn open(arena: &'a Arena, connection: &Mutex<Cache>, key: Option<&Key>) -> Result<Self> {
560623 let (section, open, touched) = replay(arena, &*lock(connection)?, key)?;
561624 Ok(Self {
562625 section,
......@@ -572,23 +635,53 @@ impl<'a> Working<'a> {
572635 requests: &mpsc::Receiver<Request>,
573636 backlog: &mut VecDeque<Request>,
574637 ) -> Next {
575 let connection = &shared.connection;
638 let connection = &shared.cache;
576639 // While another thread rebuilds the section, reads answer from this one and every
577640 // other request waits for the rebuilt section.
578641 let mut held: Option<VecDeque<Request>> = None;
642 // While a reread waits for the closed cache, reads answer from this section and edits
643 // wait for the reread, as one this section refused may underlie them.
644 let mut stale: Option<VecDeque<Request>> = None;
645 let mut attempted = web_time::Instant::now();
579646 loop {
580647 let mut burst = std::mem::take(backlog);
581 // Only a burst that took an edit, or a retry's tick, writes at its end: reads never.
648 // Only a burst that took an edit, or a retry falling due, writes at its end: reads
649 // never.
582650 let mut write = false;
651 let retry = !self.unwritten.is_empty() || closed(connection);
583652 if burst.is_empty() {
584 match receive(requests, !self.unwritten.is_empty()).await {
653 let wait = retry.then(|| RETRY.saturating_sub(attempted.elapsed()));
654 match receive(requests, wait).await {
585655 Ok(request) => burst.push_back(request),
586 Err(mpsc::RecvTimeoutError::Timeout) => write = true,
656 Err(mpsc::RecvTimeoutError::Timeout) => {}
587657 Err(mpsc::RecvTimeoutError::Disconnected) => {
588 let _ = self.flush(shared);
658 let stale_section = stale.is_some();
659 // A refusal now leaves the section holding an edit the queue lacks too.
660 #[cfg_attr(target_arch = "wasm32", allow(unused_variables))]
661 let reread = crate::working::write(
662 shared,
663 &mut self.open,
664 &mut self.unwritten,
665 !stale_section,
666 )
667 .is_err()
668 || stale_section;
669 for request in stale.into_iter().flatten() {
670 match request {
671 Request::Apply(accepted) => self.unwritten.push(accepted),
672 request => {
673 request.fail(io::ErrorKind::Other, "The section thread stopped")
674 }
675 }
676 }
589677 #[cfg(not(target_arch = "wasm32"))]
590 if !self.unwritten.is_empty() {
591 orphan(shared, std::mem::take(&mut self.unwritten));
678 if !self.unwritten.is_empty() || closed(connection) {
679 orphan(
680 shared,
681 self.open,
682 std::mem::take(&mut self.unwritten),
683 reread,
684 );
592685 }
593686 return Next::Stop;
594687 }
......@@ -599,24 +692,11 @@ impl<'a> Working<'a> {
599692 let mut later = VecDeque::new();
600693 let mut reopen = false;
601694 while !reopen && let Some(request) = burst.pop_front().or_else(|| later.pop_front()) {
695 let Some(request) = self.answer(request) else {
696 continue;
697 };
602698 if let Some(waiting) = &mut held {
603699 match request {
604 Request::Page { space, reply } => {
605 reply(self.section.page(space).map_err(Into::into))
606 }
607 Request::Pages { reply } => reply(self.section.pages().map_err(Into::into)),
608 Request::Conflicts { reply } => {
609 reply(self.section.conflicts().map_err(Into::into))
610 }
611 Request::Versions { reply } => {
612 reply(self.section.versions().map_err(Into::into))
613 }
614 Request::Version {
615 space,
616 version,
617 reply,
618 } => reply(self.section.version(space, version).map_err(Into::into)),
619 Request::Moving { space, reply } => reply(self.moving(space)),
620700 Request::Handover(to) => {
621701 let mut waiting = held.take().unwrap_or_default();
622702 waiting.extend(burst.drain(..).chain(later.drain(..)));
......@@ -630,6 +710,25 @@ impl<'a> Working<'a> {
630710 }
631711 continue;
632712 }
713 if let Some(waiting) = &mut stale {
714 match request {
715 Request::Apply(
716 accepted @ Accepted {
717 waiting: Some(_), ..
718 },
719 ) => {
720 if let Some(notice) = &accepted.waiting {
721 notice("The cache reopens once it can");
722 }
723 waiting.push_back(Request::Apply(accepted));
724 }
725 request @ Request::Reopen { .. } => waiting.push_back(request),
726 request => {
727 request.fail(io::ErrorKind::WouldBlock, "The cache reopens once it can")
728 }
729 }
730 continue;
731 }
633732 if matches!(request, Request::Flush { .. } | Request::Seal { .. })
634733 && !burst.is_empty()
635734 {
......@@ -641,7 +740,7 @@ impl<'a> Working<'a> {
641740 Request::Flush { .. } | Request::Seal { .. } | Request::Rebase { .. }
642741 ) {
643742 let failed = self.flush(shared).is_err();
644 if failed || !self.unwritten.is_empty() {
743 if failed || !self.unwritten.is_empty() || closed(connection) {
645744 // The worker retries later; the edits wait here meanwhile.
646745 request.fail(io::ErrorKind::WouldBlock, "Edits wait to be written");
647746 reopen = failed;
......@@ -650,36 +749,17 @@ impl<'a> Working<'a> {
650749 }
651750 reopen = match request {
652751 Request::Apply(accepted) => {
653 write = true;
654 self.accept(accepted)
655 }
656 Request::Page { space, reply } => {
657 reply(self.section.page(space).map_err(Into::into));
658 false
659 }
660 Request::Pages { reply } => {
661 reply(self.section.pages().map_err(Into::into));
662 false
663 }
664 Request::Conflicts { reply } => {
665 reply(self.section.conflicts().map_err(Into::into));
666 false
667 }
668 Request::Versions { reply } => {
669 reply(self.section.versions().map_err(Into::into));
670 false
671 }
672 Request::Version {
673 space,
674 version,
675 reply,
676 } => {
677 reply(self.section.version(space, version).map_err(Into::into));
678 false
679 }
680 Request::Moving { space, reply } => {
681 reply(self.moving(space));
682 false
752 // While the cache is closed, an edit nobody waits for is refused before
753 // it applies, instead of showing until a reread can drop it.
754 if accepted.waiting.is_none()
755 && let Err(error) = lock(connection).map(drop)
756 {
757 (accepted.reply)(Err(error));
758 false
759 } else {
760 write = true;
761 self.accept(accepted)
762 }
683763 }
684764 Request::Flush { reply, durable } => {
685765 #[cfg(target_arch = "wasm32")]
......@@ -715,6 +795,12 @@ impl<'a> Working<'a> {
715795 }
716796 Request::Reopen { reply } => {
717797 let _ = self.flush(shared);
798 if closed(connection) {
799 stale
800 .get_or_insert_default()
801 .push_back(Request::Reopen { reply });
802 continue;
803 }
718804 #[cfg(target_arch = "wasm32")]
719805 let reopen = inline(shared, Job::Reopen { reply });
720806 #[cfg(not(target_arch = "wasm32"))]
......@@ -726,20 +812,86 @@ impl<'a> Working<'a> {
726812 };
727813 reopen
728814 }
729 Request::Handover(_) | Request::Resume => false,
815 Request::Page { .. }
816 | Request::Pages { .. }
817 | Request::Conflicts { .. }
818 | Request::Versions { .. }
819 | Request::Version { .. }
820 | Request::Moving { .. }
821 | Request::Handover(_)
822 | Request::Resume => false,
823 };
824 }
825 // Retries fall due once a second however busy the section is; a closed cache is
826 // tried then alone, not with each edit.
827 if retry {
828 let due = attempted.elapsed() >= RETRY;
829 write = if closed(connection) {
830 due
831 } else {
832 write || due
730833 };
731834 }
732 if reopen || write && self.flush(shared).is_err() {
733 backlog.extend(
734 self.unwritten
735 .drain(..)
736 .map(Request::Apply)
737 .chain(burst.drain(..))
738 .chain(later.drain(..)),
739 );
740 return Next::Reopen;
835 if write {
836 attempted = web_time::Instant::now();
837 }
838 if let Some(waiting) = &stale {
839 // Only whether the closed commit landed is settled before the reread.
840 if write {
841 let _ =
842 crate::working::write(shared, &mut self.open, &mut self.unwritten, false);
843 }
844 if closed(connection) {
845 let held = waiting
846 .iter()
847 .filter(|request| matches!(request, Request::Apply(_)))
848 .count();
849 shared.wait(self.unwritten.len() + held);
850 continue;
851 }
852 } else if reopen || write && self.flush(shared).is_err() {
853 if closed(connection) {
854 stale = Some(VecDeque::new());
855 backlog.extend(burst.drain(..).chain(later.drain(..)));
856 continue;
857 }
858 } else {
859 continue;
860 }
861 backlog.extend(
862 self.unwritten
863 .drain(..)
864 .map(Request::Apply)
865 .chain(stale.into_iter().flatten())
866 .chain(burst.drain(..))
867 .chain(later.drain(..)),
868 );
869 // What the host asks next answers from the reread section.
870 if let (Ok(pages), Ok(reread)) = (self.section.pages(), shared.reread.lock())
871 && let Some(reread) = reread.as_ref()
872 {
873 reread(pages.into_iter().map(|(space, ..)| space).collect());
741874 }
875 return Next::Reopen;
876 }
877 }
878
879 /// Answers `request` where it reads the section; any other comes back.
880 fn answer(&mut self, request: Request) -> Option<Request> {
881 match request {
882 Request::Page { space, reply } => reply(self.section.page(space).map_err(Into::into)),
883 Request::Pages { reply } => reply(self.section.pages().map_err(Into::into)),
884 Request::Conflicts { reply } => reply(self.section.conflicts().map_err(Into::into)),
885 Request::Versions { reply } => reply(self.section.versions().map_err(Into::into)),
886 Request::Version {
887 space,
888 version,
889 reply,
890 } => reply(self.section.version(space, version).map_err(Into::into)),
891 Request::Moving { space, reply } => reply(self.moving(space)),
892 request => return Some(request),
742893 }
894 None
743895 }
744896
745897 /// Starts a thread that runs `job` and rereads the section; false when the replica is
......@@ -807,109 +959,12 @@ impl<'a> Working<'a> {
807959 }
808960 }
809961
810 /// Writes the unwritten edits in one transaction, then answers their senders. A write
811 /// that may succeed later keeps the edits with a notice, telling the first; one that never
812 /// will refuses the edit it failed on. An error means the section holds an edit the queue
813 /// never will, or the cache closed, and must be reread.
814962 fn flush(&mut self, shared: &Arc<Thread>) -> Result<()> {
815 if self.unwritten.is_empty() {
816 return Ok(());
817 }
818 let mut connection = lock(&shared.connection)?;
819 let mut ids = Vec::new();
820 let committed = connection
821 .transaction_with_behavior(TransactionBehavior::Immediate)
822 .map_err(crate::Error::from)
823 .and_then(|transaction| {
824 let batch = match self.open {
825 Some(batch) => batch,
826 None => {
827 transaction.execute("INSERT INTO batches DEFAULT VALUES", [])?;
828 transaction.last_insert_rowid()
829 }
830 };
831 for edit in &self.unwritten {
832 ids.push(queue::insert(
833 &transaction,
834 shared.key.as_ref(),
835 None,
836 batch,
837 &edit.author,
838 &edit.edit,
839 )?);
840 }
841 Ok((batch, transaction.commit().err()))
842 });
843 // The error, and the edit it refuses when it would fail again.
844 let written = match committed {
845 Ok((batch, None)) => Ok(batch),
846 Ok((batch, Some(error))) => {
847 if landed(&mut connection, ids.last().copied().unwrap_or_default())? {
848 Ok(batch)
849 } else {
850 Err((error.into(), None))
851 }
852 }
853 Err(error) => {
854 let refused = (!transient(&error)).then_some(ids.len());
855 Err((error, refused))
856 }
857 };
858 drop(connection);
859 let batch = match written {
860 Ok(batch) => batch,
861 Err((error, refused)) => {
862 let message = error.to_string();
863 let before = self.unwritten.len();
864 let mut at = 0;
865 for edit in self.unwritten.extract_if(.., |edit| {
866 at += 1;
867 edit.waiting.is_none() || refused == Some(at - 1)
868 }) {
869 (edit.reply)(Err(io::Error::other(message.clone()).into()));
870 }
871 shared.wait(self.unwritten.len());
872 if let Some(notice) = self
873 .unwritten
874 .first()
875 .and_then(|edit| edit.waiting.as_ref())
876 {
877 notice(&message);
878 }
879 return if self.unwritten.len() == before {
880 Ok(())
881 } else {
882 Err(error)
883 };
884 }
885 };
886 self.open = Some(batch);
887 #[cfg(target_arch = "wasm32")]
888 shared.receipt(
889 self.unwritten
890 .drain(..)
891 .zip(ids)
892 .map(|(edit, id)| {
893 let reply: Reply<()> =
894 Box::new(move |result| (edit.reply)(result.map(|()| id)));
895 reply
896 })
897 .collect(),
898 true,
899 );
900 #[cfg(not(target_arch = "wasm32"))]
901 {
902 for (edit, id) in self.unwritten.drain(..).zip(ids) {
903 (edit.reply)(Ok(id));
904 }
905 crate::edited(&shared.worker);
906 }
907 shared.wait(0);
908 Ok(())
963 write(shared, &mut self.open, &mut self.unwritten, true)
909964 }
910965
911966 /// Seals the open batch unless a sealed batch still waits for publication.
912 fn seal(&mut self, connection: &Mutex<Connection>) -> Result<Option<Sealed>> {
967 fn seal(&mut self, connection: &Mutex<Cache>) -> Result<Option<Sealed>> {
913968 let Some(batch) = self.open else {
914969 return Ok(None);
915970 };
......@@ -954,7 +1009,7 @@ impl<'a> Working<'a> {
9541009/// version the remote could not take gains a conflict page holding it, queued as a new edit.
9551010/// Returns the pages the remote changed.
9561011fn rebase(
957 connection: &Mutex<Connection>,
1012 connection: &Mutex<Cache>,
9581013 image: Option<Vec<u8>>,
9591014 key: Option<&Key>,
9601015) -> Result<Vec<ExGuid>> {
......@@ -1186,28 +1241,206 @@ fn transient(error: &crate::Error) -> bool {
11861241 if matches!(error.code, SystemIoFailure | DiskFull | DatabaseBusy | DatabaseLocked | CannotOpen))
11871242}
11881243
1189/// Whether the commit whose last edit is `last` reached the cache after it failed, as a cold
1190/// open reads it, a commit landing whole: the warm connection may not know of frames its WAL
1191/// holds, and must never write again. When the cache cannot be reopened, an inert connection
1192/// stays, failing every statement.
1193fn landed(connection: &mut Connection, last: u64) -> Result<bool> {
1194 let path = connection.path().map(PathBuf::from);
1195 let inert = Connection::open_in_memory()?;
1196 inert.execute_batch("PRAGMA query_only=1")?;
1197 std::mem::replace(connection, inert)
1198 .close()
1199 .map_err(|(_, error)| error)?;
1244/// Writes `unwritten` in one transaction after the open batch `open`, then answers their
1245/// senders. A write that may succeed later keeps the edits with a notice; one that never will
1246/// refuses the edit it failed on, and an edit nobody waits for is refused at once. A failed
1247/// commit leaves the cache closed, still owned, until it reopens to decide whether the commit
1248/// landed; a lost cache refuses every edit. An error means the section holds an edit the
1249/// queue never will, or the cache is lost, and must be reread.
1250fn write(
1251 shared: &Arc<Thread>,
1252 open: &mut Option<i64>,
1253 unwritten: &mut Vec<Accepted>,
1254 commit: bool,
1255) -> Result<()> {
1256 let mut cache = (shared.cache.lock()).map_err(|_| io::Error::other("Cache owner panicked"))?;
1257 if unwritten.is_empty() && !matches!(*cache, Cache::Closed { .. }) {
1258 return Ok(());
1259 }
1260 let path = shared.path.as_deref();
1261 // The ids the cache holds for the first of the edits.
1262 let mut ids = Vec::new();
1263 // The error, and the edit it refuses when it would fail again.
1264 let failed = (|| {
1265 ids = reopen(&mut cache, path, open).map_err(|error| (error, None))?;
1266 if !commit || ids.len() == unwritten.len() {
1267 return Ok(());
1268 }
1269 let Cache::Open(connection) = &mut *cache else {
1270 unreachable!("A cache that does not reopen fails");
1271 };
1272 let mut inserted = Vec::new();
1273 let mut queued = 0;
1274 let committed = connection
1275 .transaction_with_behavior(TransactionBehavior::Immediate)
1276 .map_err(crate::Error::from)
1277 .and_then(|transaction| {
1278 queued = transaction
1279 .query_row("SELECT count(*) FROM edits", [], |row| row.get::<_, i64>(0))?;
1280 let batch = match *open {
1281 Some(batch) => batch,
1282 None => {
1283 transaction.execute("INSERT INTO batches DEFAULT VALUES", [])?;
1284 transaction.last_insert_rowid()
1285 }
1286 };
1287 for edit in &unwritten[ids.len()..] {
1288 inserted.push(queue::insert(
1289 &transaction,
1290 shared.key.as_ref(),
1291 None,
1292 batch,
1293 &edit.author,
1294 &edit.edit,
1295 )?);
1296 }
1297 Ok((batch, transaction.commit().err()))
1298 });
1299 match committed {
1300 Ok((batch, None)) => *open = Some(batch),
1301 Ok((_, Some(error))) => {
1302 // The warm connection may not know of frames its WAL holds, and must never
1303 // write again; it keeps the cache owned until the owner lock can.
1304 if let Cache::Open(warm) =
1305 std::mem::replace(&mut *cache, Cache::Lost(String::new()))
1306 {
1307 *cache = Cache::Closed {
1308 last: inserted[inserted.len() - 1],
1309 ids: inserted,
1310 queued: queued.unsigned_abs(),
1311 owner: Owner::Warm(warm),
1312 };
1313 }
1314 inserted = reopen(&mut cache, path, open).map_err(|error| (error, None))?;
1315 if inserted.is_empty() {
1316 return Err((error.into(), None));
1317 }
1318 }
1319 Err(error) => {
1320 let refused = (!transient(&error)).then_some(inserted.len());
1321 return Err((error, refused));
1322 }
1323 }
1324 ids.extend(inserted);
1325 Ok(())
1326 })();
1327 let written: Vec<_> = unwritten.drain(..ids.len()).zip(ids).collect();
1328 let refused: Vec<_> = match &failed {
1329 Ok(()) => Vec::new(),
1330 Err((_, failing)) => {
1331 let lost = matches!(*cache, Cache::Lost(_));
1332 let (mut at, mut kept) = (0, 0);
1333 (unwritten.extract_if(.., |edit| {
1334 at += 1;
1335 let refuse = lost || edit.waiting.is_none() || *failing == Some(at - 1);
1336 if !refuse {
1337 kept += 1;
1338 } else if let Cache::Closed { ids, .. } = &mut *cache
1339 && kept < ids.len()
1340 {
1341 // Whether its commit landed, it is answered now; a reread shows which.
1342 ids.remove(kept);
1343 }
1344 refuse
1345 }))
1346 .collect()
1347 }
1348 };
1349 drop(cache);
1350 // Counted before anyone hears, as a host hearing may ask.
1351 shared.wait(unwritten.len());
1352 if !written.is_empty() {
1353 #[cfg(target_arch = "wasm32")]
1354 shared.receipt(
1355 written
1356 .into_iter()
1357 .map(|(edit, id)| {
1358 let reply: Reply<()> =
1359 Box::new(move |result| (edit.reply)(result.map(|()| id)));
1360 reply
1361 })
1362 .collect(),
1363 true,
1364 );
1365 #[cfg(not(target_arch = "wasm32"))]
1366 {
1367 for (edit, id) in written {
1368 (edit.reply)(Ok(id));
1369 }
1370 crate::edited(&shared.worker);
1371 }
1372 }
1373 let Err((error, _)) = failed else {
1374 return Ok(());
1375 };
1376 let message = error.to_string();
1377 let kept = refused.is_empty();
1378 for edit in refused {
1379 (edit.reply)(Err(io::Error::other(message.clone()).into()));
1380 }
1381 if let Some(notice) = unwritten.iter().find_map(|edit| edit.waiting.as_ref()) {
1382 notice(&message);
1383 }
1384 if kept { Ok(()) } else { Err(error) }
1385}
1386
1387/// Reopens the cache a failed commit closed and reads cold whether the commit landed, as one
1388/// lands whole: the ids of its edits still waiting when it did, its batch becoming `open`;
1389/// none where it did not or the cache was open. The closed cache stays owned meanwhile, and
1390/// is lost once the file is gone or no longer a database.
1391fn reopen(cache: &mut Cache, path: Option<&Path>, open: &mut Option<i64>) -> Result<Vec<u64>> {
1392 let (last, ids, owner) = match cache {
1393 Cache::Open(_) => return Ok(Vec::new()),
1394 Cache::Closed {
1395 last, ids, owner, ..
1396 } => (*last, ids, owner),
1397 Cache::Lost(error) => return Err(io::Error::other(error.clone()).into()),
1398 };
12001399 // Reopening through the browser's file system is unverified: it stays closed.
1201 let path = path
1202 .filter(|_| cfg!(not(target_arch = "wasm32")))
1203 .ok_or_else(|| io::Error::other("The cache could not confirm a write"))?;
1204 let cold = crate::cache_connection(&path)?;
1205 crate::write_ahead(&cold)?;
1206 let found: bool = cold.query_row(
1207 "SELECT EXISTS(SELECT 1 FROM edits WHERE id=?1)",
1208 [crate::signed(last)?],
1209 |row| row.get(0),
1210 )?;
1211 *connection = cold;
1212 Ok(found)
1400 let Some(path) = path else {
1401 let error = "The cache could not confirm a write";
1402 *cache = Cache::Lost(error.to_owned());
1403 return Err(io::Error::other(error).into());
1404 };
1405 let reopened = (|| -> Result<(Connection, Option<i64>)> {
1406 if let Owner::Warm(_) = owner {
1407 *owner = Owner::Owned(crate::Owned::take(path)?);
1408 }
1409 let cold = crate::connect(path)?;
1410 crate::write_ahead(&cold)?;
1411 if cold.is_readonly(rusqlite::MAIN_DB)? {
1412 return Err(io::Error::from(io::ErrorKind::ReadOnlyFilesystem).into());
1413 }
1414 let batch = cold
1415 .query_row(
1416 "SELECT batch FROM edits WHERE id=?1",
1417 [crate::signed(last)?],
1418 |row| row.get(0),
1419 )
1420 .optional()?;
1421 Ok((cold, batch))
1422 })();
1423 match reopened {
1424 Ok((cold, batch)) => {
1425 let landed = match batch {
1426 Some(batch) => {
1427 *open = Some(batch);
1428 std::mem::take(ids)
1429 }
1430 None => Vec::new(),
1431 };
1432 *cache = Cache::Open(cold);
1433 Ok(landed)
1434 }
1435 Err(error) => {
1436 use rusqlite::ErrorCode::*;
1437 let lost = !path.exists()
1438 || matches!(&error, crate::Error::Database(rusqlite::Error::SqliteFailure(error, _))
1439 if matches!(error.code, NotADatabase | DatabaseCorrupt));
1440 if lost {
1441 *cache = Cache::Lost(error.to_string());
1442 }
1443 Err(error)
1444 }
1445 }
12131446}
crates/notebook/tests/commit_fault.rs+546-55
......@@ -1,20 +1,47 @@
1//! Commits whose WAL write or sync fails, through a shim over the platform's VFS. Its own
2//! binary, as the shim becomes the process's default VFS.
1//! Commits whose WAL write or sync fails, and caches that will not reopen after one, through a
2//! shim over the platform's VFS. Its own binary, as the shim becomes the process's default VFS.
33#![cfg(unix)]
44
55use notebook::{
6 Replica,
7 session::{Event, Section},
6 EditStatus, Error, Replica,
7 session::{Event, Section, SyncState},
8};
9use onestore::{
10 PageCreation,
11 op::{Edit, Op, PageOp, SectionOp},
12};
13use std::{
14 io,
15 path::{Path, PathBuf},
16 sync::{MutexGuard, PoisonError, atomic::Ordering, mpsc},
17 time::{Duration, Instant},
818};
9use onestore::op::{Edit, Op, PageOp};
10use std::{path::Path, sync::atomic::Ordering, sync::mpsc, time::Duration};
1119
1220#[path = "support/vfs.rs"]
1321mod vfs;
1422use vfs::{
15 ARMED, KEEP_WAL, OPEN, SYNC_AFTER, SYNC_BEFORE, WAL_WRITES_AFTER_FAULT, WRITE_COMMIT, install,
23 ARMED, EXCLUSIVE, FAILING_OPENS, KEEP_WAL, SYNC_AFTER, SYNC_BEFORE, WAL_WRITES_AFTER_FAULT,
24 WRITE_COMMIT, install,
1625};
1726
27/// Holds the shim for one test, disarmed of what a failed test left.
28fn exclusive() -> MutexGuard<'static, ()> {
29 install();
30 let exclusive = EXCLUSIVE.lock().unwrap_or_else(PoisonError::into_inner);
31 ARMED.store(0, Ordering::SeqCst);
32 FAILING_OPENS.store(0, Ordering::SeqCst);
33 exclusive
34}
35
36/// A section file in `root` and its cache, which the returned path names.
37fn seed(root: &Path) -> PathBuf {
38 let source = onestore::create_section("Fault.one", "Original", "Fixture").unwrap();
39 std::fs::write(root.join("Fault.one"), &source).unwrap();
40 let cache = root.join("replica.sqlite");
41 drop(Replica::create(&cache, &source).unwrap());
42 cache
43}
44
1845fn colour(space: onestore::ExGuid, colour: u32) -> Edit {
1946 Edit {
2047 at: 133_000_000_000_000_000 + u64::from(colour),
......@@ -25,17 +52,49 @@ fn colour(space: onestore::ExGuid, colour: u32) -> Edit {
2552 }
2653}
2754
55/// The queued edits a cold open of `cache` reads, with the colour each gives the page.
56fn colours(cache: &Path) -> Vec<(u64, u32)> {
57 Replica::open(cache)
58 .unwrap()
59 .pending()
60 .unwrap()
61 .iter()
62 .map(|edit| match &edit.edit.ops[..] {
63 [
64 Op::Page {
65 op: PageOp::Color(Some(colour)),
66 ..
67 },
68 ] => (edit.id, *colour),
69 ops => panic!("{ops:?}"),
70 })
71 .collect()
72}
73
74fn until(section: &Section, seen: impl Fn(&Event) -> bool) {
75 let deadline = Instant::now() + Duration::from_secs(10);
76 while !section.events().iter().any(&seen) {
77 assert!(Instant::now() < deadline);
78 std::thread::sleep(Duration::from_millis(10));
79 }
80}
81
82fn would_block<T>(result: Result<T, Error>) -> bool {
83 matches!(result, Err(Error::Io(error)) if error.kind() == io::ErrorKind::WouldBlock)
84}
85
2886/// What the section answered for the edit committed under `fault` and the one after it,
2987/// whether a failed write was reported, and the queue as the section saw it.
3088struct Outcome {
3189 replies: [Result<u64, String>; 2],
3290 failed: bool,
33 pending: Option<Vec<u64>>,
91 pending: Vec<u64>,
3492}
3593
3694/// Seeds the WAL with one edit, so the armed commit's sync is the only one it makes, then
37/// commits under `fault` and once more after it; with `crash`, leaves the cache open.
38fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
95/// commits under `fault`, the cache failing to reopen `opens` times, and once more after it;
96/// with `crash`, leaves the cache open.
97fn scenario(root: &Path, fault: u8, opens: usize, crash: bool) -> Outcome {
3998 install();
4099 let replica = Replica::open(root.join("replica.sqlite")).unwrap();
41100 let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap();
......@@ -54,9 +113,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
54113 };
55114 apply(1).unwrap();
56115 WAL_WRITES_AFTER_FAULT.store(0, Ordering::SeqCst);
116 FAILING_OPENS.store(opens, Ordering::SeqCst);
57117 ARMED.store(fault, Ordering::SeqCst);
58118 let first = apply(2);
59119 assert_eq!(ARMED.load(Ordering::SeqCst), 0, "fault {fault} never fired");
120 assert_eq!(
121 FAILING_OPENS.load(Ordering::SeqCst),
122 0,
123 "the cache never reopened"
124 );
60125 assert_eq!(
61126 WAL_WRITES_AFTER_FAULT.load(Ordering::SeqCst),
62127 0,
......@@ -69,8 +134,10 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
69134 .any(|event| matches!(event, Event::Failed(_)));
70135 let pending = section
71136 .pending()
72 .ok()
73 .map(|pending| pending.iter().map(|edit| edit.id).collect());
137 .unwrap()
138 .iter()
139 .map(|edit| edit.id)
140 .collect();
74141 if crash {
75142 std::mem::forget(section);
76143 } else {
......@@ -85,14 +152,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
85152
86153#[test]
87154fn commit_fault_child() {
88 let (Some(root), Some(fault)) = (
155 let (Some(root), Some(fault), Some(opens)) = (
89156 std::env::var_os("COMMIT_FAULT_CHILD"),
90157 std::env::var("COMMIT_FAULT").ok(),
158 std::env::var("COMMIT_FAULT_OPENS").ok(),
91159 ) else {
92160 return;
93161 };
94162 let root = Path::new(&root);
95 let outcome = scenario(root, fault.parse().unwrap(), true);
163 let outcome = scenario(root, fault.parse().unwrap(), opens.parse().unwrap(), true);
96164 let ids = |result: &Result<u64, String>| result.as_ref().map_or(0, |id| *id);
97165 std::fs::write(
98166 root.join("outcome"),
......@@ -110,68 +178,491 @@ fn commit_fault_child() {
110178
111179#[test]
112180fn a_failed_commit_is_decided_by_a_cold_read_and_never_replayed_twice() {
113 install();
114 // (fault, whether the edit reached the cache, whether the cache stays closed)
115 for (fault, landed, closed) in [
116 (SYNC_BEFORE, false, false),
117 (SYNC_AFTER, false, false),
118 (WRITE_COMMIT, false, false),
119 (SYNC_AFTER | KEEP_WAL, true, false),
120 (SYNC_BEFORE | OPEN, false, true),
181 let _exclusive = exclusive();
182 // (fault, whether the edit reached the cache, how many reopens fail)
183 for (fault, landed, opens) in [
184 (SYNC_BEFORE, false, 0),
185 (SYNC_AFTER, false, 0),
186 (WRITE_COMMIT, false, 0),
187 (SYNC_AFTER | KEEP_WAL, true, 0),
188 (SYNC_BEFORE, false, 2),
189 (SYNC_AFTER | KEEP_WAL, true, 2),
121190 ] {
122191 for crash in [false, true] {
123192 let directory = tempfile::tempdir().unwrap();
124193 let root = directory.path();
125 let source = onestore::create_section("Fault.one", "Original", "Fixture").unwrap();
126 std::fs::write(root.join("Fault.one"), &source).unwrap();
127 drop(Replica::create(root.join("replica.sqlite"), &source).unwrap());
194 let cache = seed(root);
128195 let (first, second, failed) = if crash {
129196 let output = std::process::Command::new(std::env::current_exe().unwrap())
130197 .args(["--exact", "commit_fault_child", "--nocapture"])
131198 .env("COMMIT_FAULT_CHILD", root)
132199 .env("COMMIT_FAULT", fault.to_string())
200 .env("COMMIT_FAULT_OPENS", opens.to_string())
133201 .output()
134202 .unwrap();
135203 assert!(output.status.success(), "{output:?}");
136204 let outcome = std::fs::read_to_string(root.join("outcome")).unwrap();
137205 let mut fields = outcome.split(' ');
138 let mut id = || Some(fields.next()?.parse::<u64>().unwrap()).filter(|id| *id > 0);
206 let mut id = || fields.next().unwrap().parse::<u64>().unwrap();
139207 let (first, second) = (id(), id());
140208 (first, second, fields.next() == Some("true"))
141209 } else {
142 let outcome = scenario(root, fault, false);
143 let [first, second] = outcome.replies;
144 if !closed {
145 let ids = [first.clone().unwrap(), second.clone().unwrap()];
146 assert_eq!(&outcome.pending.unwrap()[1..], ids);
147 } else {
148 assert!(outcome.pending.is_none());
149 }
150 (first.ok(), second.ok(), outcome.failed)
210 let outcome = scenario(root, fault, opens, false);
211 let ids = outcome.replies.map(Result::unwrap);
212 assert_eq!(outcome.pending[1..], ids);
213 (ids[0], ids[1], outcome.failed)
151214 };
152 assert_eq!(failed, !landed, "fault {fault}");
153 let cold = Replica::open(root.join("replica.sqlite")).unwrap();
154 let colours: Vec<_> = cold
155 .pending()
215 // An edit waiting for the cache to reopen is reported, as one that did not land.
216 assert_eq!(failed, !landed || opens > 0, "fault {fault}, {opens} opens");
217 assert_eq!(
218 colours(&cache)[1..],
219 [(first, 2), (second, 3)],
220 "fault {fault}, {opens} opens"
221 );
222 }
223 }
224}
225
226#[test]
227fn edits_wait_while_the_cache_cannot_reopen_and_land_once() {
228 let _exclusive = exclusive();
229 for close in [false, true] {
230 let directory = tempfile::tempdir().unwrap();
231 let root = directory.path();
232 let cache = seed(root);
233 let section = Section::resume(
234 root.join("Fault.one"),
235 Replica::open(&cache).unwrap(),
236 true,
237 || {},
238 )
239 .unwrap();
240 let space = section.pages().unwrap()[0].0;
241 let (replied, replies) = mpsc::channel();
242 let apply = |value| {
243 let replied = replied.clone();
244 section
245 .apply_with("Fixture", colour(space, value), move |result| {
246 replied
247 .send(result.map_err(|error| error.to_string()))
248 .unwrap();
249 })
250 .unwrap();
251 };
252 let reply = || {
253 replies
254 .recv_timeout(Duration::from_secs(10))
156255 .unwrap()
157 .iter()
158 .map(|edit| match &edit.edit.ops[..] {
159 [
160 Op::Page {
161 op: PageOp::Color(Some(colour)),
162 ..
163 },
164 ] => (edit.id, *colour),
165 ops => panic!("{ops:?}"),
256 .unwrap()
257 };
258 apply(1);
259 let seeded = reply();
260 FAILING_OPENS.store(1000, Ordering::SeqCst);
261 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
262 apply(2);
263 until(&section, |event| matches!(event, Event::Failed(_)));
264 // The cache is tried again once a second.
265 let left = FAILING_OPENS.load(Ordering::SeqCst);
266 std::thread::sleep(Duration::from_secs(3));
267 let tries = left - FAILING_OPENS.load(Ordering::SeqCst);
268 // Never faster than once a second; as slow as the machine is busy.
269 assert!((1..=4).contains(&tries), "{tries} tries in 3 s");
270 apply(3);
271 // A caller blocking on its edit learns at once, and the waiting edits stay.
272 assert!(would_block(
273 section.replica().apply("Fixture", colour(space, 4))
274 ));
275 // A sync step meets the closed cache and the worker carries on.
276 section.wake();
277 until(
278 &section,
279 |event| matches!(event, Event::Failed(error) if error == "The cache reopens once it can"),
280 );
281 // Each second the cache is tried again, and the edits still wait.
282 until(&section, |event| matches!(event, Event::Failed(_)));
283 assert!(replies.try_recv().is_err());
284 assert_eq!(section.page(space).unwrap().color, Some(3));
285 assert!(would_block(section.written()));
286 assert!(would_block(section.pending()));
287 assert_eq!(notebook::unwritten(), 2);
288 let status = section.sync_status().unwrap();
289 // The seed, queued before the commit that failed, and the two waiting.
290 assert_eq!((status.state(), status.queued), (SyncState::Failed, 3));
291 // Nobody else takes the cache meanwhile, nor once the section closes.
292 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
293 let open = if close {
294 section.close().unwrap();
295 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
296 None
297 } else {
298 Some(section)
299 };
300 FAILING_OPENS.store(0, Ordering::SeqCst);
301 let ids = [reply(), reply()];
302 if let Some(section) = open {
303 section.wake();
304 until(&section, |event| {
305 matches!(event, Event::Attempt { id, status: EditStatus::Published { .. } }
306 if *id == ids[1])
307 });
308 section.close().unwrap();
309 let arena = onestore::Arena::default();
310 let stored =
311 onestore::Section::open(&arena, std::fs::read(root.join("Fault.one")).unwrap())
312 .unwrap();
313 assert_eq!(stored.page(space).unwrap().color, Some(3));
314 assert!(colours(&cache).is_empty());
315 } else {
316 // Its cache is released once the edits are written.
317 let deadline = Instant::now() + Duration::from_secs(10);
318 while Replica::open(&cache).is_err() {
319 assert!(Instant::now() < deadline);
320 std::thread::sleep(Duration::from_millis(10));
321 }
322 assert_eq!(colours(&cache), [(seeded, 1), (ids[0], 2), (ids[1], 3)]);
323 }
324 assert_eq!(notebook::unwritten(), 0);
325 }
326}
327
328#[test]
329fn a_lost_cache_refuses_the_edits_that_wait() {
330 let _exclusive = exclusive();
331 for corrupt in [false, true] {
332 let directory = tempfile::tempdir().unwrap();
333 let root = directory.path();
334 let cache = seed(root);
335 let section = Section::resume(
336 root.join("Fault.one"),
337 Replica::open(&cache).unwrap(),
338 true,
339 || {},
340 )
341 .unwrap();
342 let space = section.pages().unwrap()[0].0;
343 let (replied, replies) = mpsc::channel();
344 let apply = |value| {
345 let replied = replied.clone();
346 section
347 .apply_with("Fixture", colour(space, value), move |result| {
348 replied
349 .send(result.map_err(|error| error.to_string()))
350 .unwrap();
166351 })
167 .collect();
168 if closed {
169 assert_eq!((first, second), (None, None));
170 assert_eq!(colours.len(), 1, "fault {fault}");
171 } else {
172 let (first, second) = (first.unwrap(), second.unwrap());
173 assert_eq!(&colours[1..], [(first, 2), (second, 3)], "fault {fault}");
352 .unwrap();
353 };
354 let reply = || replies.recv_timeout(Duration::from_secs(10)).unwrap();
355 apply(1);
356 reply().unwrap();
357 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
358 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
359 apply(2);
360 until(&section, |event| matches!(event, Event::Failed(_)));
361 for suffix in ["-wal", "-shm"] {
362 let mut file = cache.clone().into_os_string();
363 file.push(suffix);
364 let _ = std::fs::remove_file(file);
365 }
366 if corrupt {
367 std::fs::write(&cache, [0xab; 4096]).unwrap();
368 } else {
369 std::fs::remove_file(&cache).unwrap();
370 }
371 FAILING_OPENS.store(0, Ordering::SeqCst);
372 assert!(reply().is_err());
373 assert_eq!(notebook::unwritten(), 0);
374 let pending = section.pending();
375 assert!(pending.is_err() && !would_block(pending));
376 apply(3);
377 assert!(reply().is_err());
378 let status = section.sync_status().unwrap();
379 assert_eq!((status.state(), status.queued), (SyncState::Failed, 0));
380 // The sync worker meets the lost cache and stops.
381 std::thread::sleep(Duration::from_millis(100));
382 section.events();
383 section.wake();
384 until(&section, |event| matches!(event, Event::Failed(_)));
385 assert!(section.close().is_err());
386 }
387}
388
389#[test]
390fn a_closed_cache_stays_owned_and_its_landed_commit_counts_once() {
391 let _exclusive = exclusive();
392 for close in [false, true] {
393 let directory = tempfile::tempdir().unwrap();
394 let root = directory.path();
395 let cache = seed(root);
396 let section = Section::resume(
397 root.join("Fault.one"),
398 Replica::open(&cache).unwrap(),
399 true,
400 || {},
401 )
402 .unwrap();
403 let space = section.pages().unwrap()[0].0;
404 let (replied, replies) = mpsc::channel::<Result<u64, String>>();
405 let reply = || {
406 replies
407 .recv_timeout(Duration::from_secs(10))
408 .unwrap()
409 .unwrap()
410 };
411 let apply = |value| {
412 let replied = replied.clone();
413 section
414 .apply_with("Fixture", colour(space, value), move |result| {
415 replied
416 .send(result.map_err(|error| error.to_string()))
417 .unwrap();
418 })
419 .unwrap();
420 };
421 apply(1);
422 let seeded = reply();
423 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
424 ARMED.store(SYNC_AFTER | KEEP_WAL, Ordering::SeqCst);
425 // A caller blocking on an edit whose commit fails hears at once, though it landed.
426 let asked = Instant::now();
427 assert!(
428 section
429 .replica()
430 .apply("Fixture", colour(space, 2))
431 .is_err()
432 );
433 assert!(asked.elapsed() < Duration::from_secs(5));
434 // Waits after the commit the closed cache holds, in the same batch.
435 apply(3);
436 until(&section, |event| matches!(event, Event::Failed(_)));
437 // Nobody else takes the cache to publish and acknowledge that commit meanwhile.
438 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
439 let open = if close {
440 section.close().unwrap();
441 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
442 None
443 } else {
444 Some(section)
445 };
446 FAILING_OPENS.store(0, Ordering::SeqCst);
447 let third = reply();
448 if let Some(section) = open {
449 // Reread once the cache reopens, the section is the queue.
450 let deadline = Instant::now() + Duration::from_secs(10);
451 while !section.pending().is_ok_and(|pending| pending.len() == 3) {
452 assert!(Instant::now() < deadline);
453 std::thread::sleep(Duration::from_millis(10));
454 }
455 assert_eq!(section.page(space).unwrap().color, Some(3));
456 section.close().unwrap();
457 } else {
458 let deadline = Instant::now() + Duration::from_secs(10);
459 while Replica::open(&cache).is_err() {
460 assert!(Instant::now() < deadline);
461 std::thread::sleep(Duration::from_millis(10));
174462 }
175463 }
464 let stored = colours(&cache);
465 assert_eq!(stored.len(), 3, "{stored:?}");
466 assert_eq!(
467 (stored[0], stored[1].1, stored[2]),
468 ((seeded, 1), 2, (third, 3))
469 );
176470 }
177471}
472
473#[test]
474fn closed_crash_child() {
475 let (Some(root), Some(fault)) = (
476 std::env::var_os("CLOSED_CRASH_CHILD"),
477 std::env::var("COMMIT_FAULT").ok(),
478 ) else {
479 return;
480 };
481 install();
482 let root = Path::new(&root);
483 let replica = Replica::open(root.join("replica.sqlite")).unwrap();
484 let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap();
485 let space = section.pages().unwrap()[0].0;
486 let (replied, replies) = mpsc::channel();
487 let replied = move |result: Result<u64, Error>| replied.send(result.is_ok()).unwrap();
488 let apply = |value, replied: Box<dyn FnOnce(Result<u64, Error>) + Send>| {
489 section
490 .apply_with("Fixture", colour(space, value), replied)
491 .unwrap();
492 };
493 apply(1, Box::new(replied));
494 assert!(replies.recv_timeout(Duration::from_secs(10)).unwrap());
495 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
496 ARMED.store(fault.parse().unwrap(), Ordering::SeqCst);
497 apply(2, Box::new(drop));
498 until(&section, |event| matches!(event, Event::Failed(_)));
499 apply(3, Box::new(drop));
500 std::thread::sleep(Duration::from_millis(100));
501 // Ends as a crash would, the cache still closed.
502 std::process::exit(0);
503}
504
505#[test]
506fn a_crash_while_the_cache_is_closed_leaves_what_landed() {
507 for (fault, landed) in [(SYNC_BEFORE, false), (SYNC_AFTER | KEEP_WAL, true)] {
508 let directory = tempfile::tempdir().unwrap();
509 let root = directory.path();
510 let cache = seed(root);
511 let output = std::process::Command::new(std::env::current_exe().unwrap())
512 .args(["--exact", "closed_crash_child", "--nocapture"])
513 .env("CLOSED_CRASH_CHILD", root)
514 .env("COMMIT_FAULT", fault.to_string())
515 .output()
516 .unwrap();
517 assert!(output.status.success(), "{output:?}");
518 let colours: Vec<_> = colours(&cache)
519 .into_iter()
520 .map(|(_, colour)| colour)
521 .collect();
522 assert_eq!(
523 colours,
524 if landed { vec![1, 2] } else { vec![1] },
525 "fault {fault}"
526 );
527 }
528}
529
530/// A section, offline, over a seeded cache in `root`, with its first page's space.
531fn opened(root: &Path) -> (PathBuf, Section, onestore::ExGuid) {
532 let cache = seed(root);
533 let section = Section::resume(
534 root.join("Fault.one"),
535 Replica::open(&cache).unwrap(),
536 true,
537 || {},
538 )
539 .unwrap();
540 let space = section.pages().unwrap()[0].0;
541 (cache, section, space)
542}
543
544#[test]
545fn edits_wait_for_the_reread_a_refused_edit_needs_while_the_cache_is_closed() {
546 let _exclusive = exclusive();
547 for close in [false, true] {
548 let directory = tempfile::tempdir().unwrap();
549 let root = directory.path();
550 let (cache, section, space) = opened(root);
551 let (replied, replies) = mpsc::channel::<(u32, Result<u64, String>)>();
552 let apply = |space, value| {
553 let replied = replied.clone();
554 section
555 .apply_with("Fixture", colour(space, value), move |result| {
556 let result = result.map_err(|error| error.to_string());
557 replied.send((value, result)).unwrap();
558 })
559 .unwrap();
560 };
561 let reply = || replies.recv_timeout(Duration::from_secs(10)).unwrap();
562 apply(space, 1);
563 let seeded = reply().1.unwrap();
564 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
565 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
566 // A page a caller blocks on is refused, its commit never landing.
567 let creation = PageCreation::new(None, Some("Refused"), "Fixture").unwrap();
568 let made = creation.space();
569 let create = Edit {
570 at: 133_000_000_000_000_100,
571 ops: vec![Op::Section(SectionOp::Create(creation))],
572 };
573 assert!(section.replica().apply("Fixture", create).is_err());
574 // Until the cache reopens to reread the section, edits wait to apply, one on the
575 // refused page among them.
576 apply(made, 2);
577 apply(space, 3);
578 std::thread::sleep(Duration::from_millis(1500));
579 assert!(replies.try_recv().is_err());
580 assert_eq!(section.page(space).unwrap().color, Some(1));
581 assert!(would_block(section.written()));
582 assert_eq!(notebook::unwritten(), 2);
583 let open = if close {
584 section.close().unwrap();
585 None
586 } else {
587 Some(section)
588 };
589 FAILING_OPENS.store(0, Ordering::SeqCst);
590 // A host reading the section steadily, as one typing does, keeps no retry back.
591 let deadline = Instant::now() + Duration::from_secs(10);
592 let mut answered = Vec::new();
593 while answered.len() < 2 {
594 assert!(Instant::now() < deadline);
595 if let Some(section) = &open {
596 let _ = section.page(space);
597 }
598 answered.extend(replies.recv_timeout(Duration::from_millis(300)));
599 }
600 let mut answered: [_; 2] = answered.try_into().unwrap();
601 answered.sort_by_key(|(value, _)| *value);
602 let [(_, dependent), (_, kept)] = answered;
603 assert!(dependent.is_err(), "{dependent:?}");
604 let kept = kept.unwrap();
605 if let Some(section) = open {
606 // The reread tells the host to reload what it shows.
607 until(
608 &section,
609 |event| matches!(event, Event::Changed(spaces) if spaces.contains(&made)),
610 );
611 assert!(
612 section
613 .pages()
614 .unwrap()
615 .iter()
616 .all(|(listed, ..)| *listed != made)
617 );
618 assert_eq!(section.page(space).unwrap().color, Some(3));
619 section.close().unwrap();
620 }
621 let deadline = Instant::now() + Duration::from_secs(10);
622 while Replica::open(&cache).is_err_and(|error| error.busy()) {
623 assert!(Instant::now() < deadline);
624 std::thread::sleep(Duration::from_millis(10));
625 }
626 assert_eq!(colours(&cache), [(seeded, 1), (kept, 3)]);
627 }
628}
629
630#[test]
631fn a_cache_whose_owner_lock_cannot_be_taken_stays_held_by_its_warm_connection() {
632 let _exclusive = exclusive();
633 let directory = tempfile::tempdir().unwrap();
634 let root = directory.path();
635 let (cache, section, space) = opened(root);
636 let (replied, replies) = mpsc::channel();
637 let apply = |value| {
638 let replied = replied.clone();
639 section
640 .apply_with("Fixture", colour(space, value), move |result| {
641 replied
642 .send(result.map_err(|error| error.to_string()))
643 .unwrap();
644 })
645 .unwrap();
646 };
647 let reply = || {
648 replies
649 .recv_timeout(Duration::from_secs(10))
650 .unwrap()
651 .unwrap()
652 };
653 apply(1);
654 let seeded = reply();
655 let mut lock = cache.clone().into_os_string();
656 lock.push("-lock");
657 std::fs::create_dir(&lock).unwrap();
658 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
659 apply(2);
660 until(&section, |event| matches!(event, Event::Failed(_)));
661 std::thread::sleep(Duration::from_millis(1500));
662 assert!(replies.try_recv().is_err());
663 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
664 std::fs::remove_dir(&lock).unwrap();
665 let second = reply();
666 section.close().unwrap();
667 assert_eq!(colours(&cache), [(seeded, 1), (second, 2)]);
668}
crates/notebook/tests/support/vfs.rs+24-7
......@@ -1,6 +1,6 @@
11//! A VFS over the platform's, made the process's default by `install`, that fails what a
2//! test arms: once, the WAL's sync or commit frame or the next database open; or every WAL
3//! write of the database `FAILING` names, for as long as it names it.
2//! test arms: once, the WAL's sync or commit frame; the replica's next `FAILING_OPENS` opens;
3//! or every WAL write of the database `FAILING` names, for as long as it names it.
44#![allow(dead_code)]
55
66use rusqlite::ffi;
......@@ -19,8 +19,6 @@ pub const SYNC_BEFORE: u8 = 1;
1919pub const SYNC_AFTER: u8 = 2;
2020/// The commit frame's header is never written.
2121pub const WRITE_COMMIT: u8 = 4;
22/// The next open of a database fails, as the cold reopen after a failed commit.
23pub const OPEN: u8 = 8;
2422/// After a fault, the next sync fails, as the checkpoint closing the warm connection makes,
2523/// which keeps the WAL.
2624pub const KEEP_WAL: u8 = 16;
......@@ -29,12 +27,15 @@ pub static ARMED: AtomicU8 = AtomicU8::new(0);
2927/// Between a fault and the next database open, as the replica reopens the cache cold.
3028static FIRED: AtomicBool = AtomicBool::new(false);
3129pub static WAL_WRITES_AFTER_FAULT: AtomicUsize = AtomicUsize::new(0);
30/// How many database opens on a replica's own threads still fail, as the cold reopen after a
31/// failed commit can, of the database `FAILING` names where it names one; a test's opens never do.
32pub static FAILING_OPENS: AtomicUsize = AtomicUsize::new(0);
3233static REAL: OnceLock<usize> = OnceLock::new();
3334/// The platform's WAL file methods, the shim's copy of them, and the copy failing every write.
3435static WAL: OnceLock<(usize, usize, usize)> = OnceLock::new();
3536/// While set, every write to this database's WAL fails, for WALs opened while it was set.
3637pub static FAILING: Mutex<Option<PathBuf>> = Mutex::new(None);
37/// Held by a test for as long as it sets `FAILING`, which one test sets at a time.
38/// Held by a test for as long as it arms a fault, which one test does at a time.
3839pub static EXCLUSIVE: Mutex<()> = Mutex::new(());
3940
4041fn fire(fault: u8) -> bool {
......@@ -58,8 +59,24 @@ unsafe extern "C" fn open(
5859) -> c_int {
5960 if flags & ffi::SQLITE_OPEN_MAIN_DB != 0 {
6061 FIRED.store(false, Ordering::SeqCst);
61 if fire(OPEN) {
62 FIRED.store(false, Ordering::SeqCst);
62 let named = unsafe { CStr::from_ptr(name) }
63 .to_string_lossy()
64 .into_owned();
65 let chosen = FAILING
66 .lock()
67 .unwrap()
68 .as_ref()
69 .is_none_or(|database| named.starts_with(&*database.to_string_lossy()));
70 if chosen
71 && std::thread::current()
72 .name()
73 .is_some_and(|name| name.starts_with("onestore-"))
74 && FAILING_OPENS
75 .try_update(Ordering::SeqCst, Ordering::SeqCst, |opens| {
76 opens.checked_sub(1)
77 })
78 .is_ok()
79 {
6380 unsafe { (*file).pMethods = std::ptr::null() };
6481 return ffi::SQLITE_CANTOPEN;
6582 }
crates/snowbound/src/manage.rs+57-12
......@@ -436,12 +436,7 @@ impl State {
436436 state.vanishing.remove(&location);
437437 let waiting = (state.session.as_ref())
438438 .filter(|session| session.library.location == location)
439 .is_some_and(|session| {
440 session
441 .section
442 .pending()
443 .is_ok_and(|pending| !pending.is_empty())
444 });
439 .is_some_and(|session| unsaved(&session.section) > 0);
445440 let listed = (state.notebooks.iter()).find(|open| open.location == location);
446441 if let Some(library) = listed.cloned().filter(|_| gone && !waiting) {
447442 state.close_notebook(&library);
......@@ -583,10 +578,8 @@ impl State {
583578 .filter(|library| library.in_icloud())
584579 .cloned()
585580 .collect();
586 let waiting = (self.sessions()).filter(|session| {
587 session.library.in_icloud()
588 && (session.section.pending()).is_ok_and(|pending| !pending.is_empty())
589 });
581 let waiting = (self.sessions())
582 .filter(|session| session.library.in_icloud() && unsaved(&session.section) > 0);
590583 for session in waiting {
591584 let Some(folder) = platform::settings_dir().map(|folder| folder.join("Recovery"))
592585 else {
......@@ -653,8 +646,7 @@ impl State {
653646 self.persist().unwrap_or_else(|error| eprintln!("{error}"));
654647 let open = (self.session.as_ref())
655648 .filter(|session| Arc::ptr_eq(&session.library, &library))
656 .and_then(|session| session.section.pending().ok())
657 .map_or(0, |pending| pending.len() as u64);
649 .map_or(0, |session| unsaved(&session.section));
658650 let waiting = (library.background.iter())
659651 .flat_map(|background| background.status())
660652 .map(|(_, sync)| sync.queued)
......@@ -1211,6 +1203,12 @@ fn art_ops(page: &Page, art: Vec<Image>) -> Vec<PageOp> {
12111203 ops
12121204}
12131205
1206/// The changes `section` holds that its file does not yet, as many as can be told, and at least
1207/// one where none can.
1208fn unsaved(section: &notebook::session::Section) -> u64 {
1209 section.sync_status().map_or(1, |sync| sync.queued)
1210}
1211
12141212#[cfg(test)]
12151213mod tests {
12161214 use super::*;
......@@ -1225,6 +1223,53 @@ mod tests {
12251223 creation.dated(&date, &time).unwrap()
12261224 }
12271225
1226 #[test]
1227 fn unsaved_counts_what_a_closed_cache_holds() {
1228 use std::sync::atomic::Ordering;
1229 crate::vfs::install();
1230 let _exclusive =
1231 (crate::vfs::EXCLUSIVE.lock()).unwrap_or_else(std::sync::PoisonError::into_inner);
1232 let temporary =
1233 std::env::temp_dir().join(format!("snowbound-unsaved-{}", std::process::id()));
1234 let _ = notebook::fs::remove_dir_all(&temporary);
1235 notebook::fs::create_dir_all(&temporary).unwrap();
1236 let file = temporary.join("Unsaved.one");
1237 let cache = temporary.join("unsaved.sqlite");
1238 let source = onestore::create_section("Unsaved.one", "Original", AUTHOR).unwrap();
1239 notebook::fs::write(&file, &source).unwrap();
1240 let open = || {
1241 let replica = notebook::Replica::open(&cache).unwrap();
1242 notebook::session::Section::resume(&file, replica, true, || {}).unwrap()
1243 };
1244 drop(notebook::Replica::create(&cache, &source).unwrap());
1245 let section = open();
1246 let space = section.pages().unwrap()[0].0;
1247 let colour = |colour| Edit {
1248 at: crate::filetime(),
1249 ops: vec![Op::Page {
1250 space,
1251 op: PageOp::Color(Some(colour)),
1252 }],
1253 };
1254 section.replica().apply(AUTHOR, colour(1)).unwrap();
1255 section.close().unwrap();
1256 // The cache's next commit fails, and it cannot reopen; the edit is refused.
1257 *crate::vfs::FAILING.lock().unwrap() = Some(cache.canonicalize().unwrap());
1258 crate::vfs::FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
1259 let section = open();
1260 assert_eq!(unsaved(&section), 1);
1261 assert!(section.replica().apply(AUTHOR, colour(2)).is_err());
1262 assert!(section.pending().is_err());
1263 assert_eq!(unsaved(&section), 1);
1264 *crate::vfs::FAILING.lock().unwrap() = None;
1265 crate::vfs::FAILING_OPENS.store(0, Ordering::SeqCst);
1266 section.close().unwrap();
1267 while notebook::Replica::open(&cache).is_err() {
1268 std::thread::sleep(std::time::Duration::from_millis(10));
1269 }
1270 let _ = notebook::fs::remove_dir_all(&temporary);
1271 }
1272
12281273 #[test]
12291274 fn repeated_section_renames_follow_identity_and_keep_the_latest_label() {
12301275 let temporary =