authoravatarclover caruso<git@paperclover.net>2026-10-16 13:52:08-07:00
committeravatarclover caruso<git@paperclover.net>2026-10-19 17:26:14-07:00
logd2b738a4ff1796471a1f7581f203e652392fc309
tree4f49e7e6a4a645a74c64b567e4cf129848d60768
parent132fa1379f8ba53b504a5bdda15a9cc58d5f44cd
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU

fix(notebook): keep edits waiting while the cache cannot reopen

After a failed COMMIT, a cold reopen decides whether the commit landed. When that reopen fails for now (too many open files, a busy cache, memory pressure), the cache stays closed instead of failing the section: the edits wait in memory, the reopen is retried once a second however busy the section is, and the commit is decided once it succeeds, so its edits land exactly once. The closed cache stays owned. The warm connection, never used again, keeps SQLite's lock until the replica holds <cache>-lock and lists the cache file for the process in its stead, and every cache open fails busy while either is held; the check runs once the opener holds SQLite's lock, so no second owner can publish and acknowledge the undecided commit and have it queued again. A replica dropped meanwhile keeps retrying and keeps the cache owned. An edit a caller blocks on is refused at once, as under #136. The reread that refusal needs waits for the cache: page reads answer from the section as it was, later edits wait to apply to the reread section, and once the cache reopens only the closed commit is decided before the edits apply again, as #136 does, so an edit on a refused page is refused rather than queued. The session reports the section's pages as changed after such a reread. Meanwhile reads of the cache answer WouldBlock and the sync worker carries on; sync_status reports the edits queued and waiting and a failure instead of erring, and the desktop counts unsaved changes through it. Only a cache file that is gone, or no longer a database, is lost: every waiting edit is refused with the error and the section stays closed. A cold open that comes back read-only is retried. - Edits made while Snowbound cannot reopen its cache after a failed save now keep waiting and save once it can. Fixes #148 Assisted-by: claude-opus-5.5

9 files changed, 1301 insertions(+), 334 deletions(-)

crates/notebook/README.md+9-3
...@@ -53,11 +53,17 @@ connection holds exclusive ownership between transactions, and a second open fai...@@ -53,11 +53,17 @@ connection holds exclusive ownership between transactions, and a second open fai
53busy. No network wait occurs in a local edit. An edit a session submitted that the cache53busy. No network wait occurs in a local edit. An edit a session submitted that the cache
54cannot write for now (a full disk, an I/O error) stays applied in memory and is written54cannot write for now (a full disk, an I/O error) stays applied in memory and is written
55again, in order, each second; meanwhile `written` returns `WouldBlock`. A replica dropped55again, in order, each second; meanwhile `written` returns `WouldBlock`. A replica dropped
56meanwhile keeps its cache open and its edits retrying until they are written or the cache is56meanwhile keeps its cache owned and its edits retrying until they are written or the cache is
57deleted, and only then releases it; `notebook::unwritten()` counts the edits waiting in the57deleted, and only then releases it; `notebook::unwritten()` counts the edits waiting in the
58process, for a host to ask before it quits. After a commit fails, a fresh connection58process, for a host to ask before it quits. After a commit fails, a fresh connection
59decides whether it landed. An edit the cache could never write is refused alone, and the59decides whether it landed. Until the cache reopens (too many open files, a busy cache), the
60section thread rereads the queue.60edits wait the same way, reads of the cache answer `WouldBlock`, and the cache stays owned:
61the replica locks `<cache>-lock` and lists the cache for the process, and every open fails
62busy while it does. A cache gone or no longer a database refuses every edit waiting, as an
63error, and stays closed. An edit a caller blocks on, or one the cache could never write, is
64refused at once, and the section thread rereads the queue; while the cache is closed, that
65reread waits for it, page reads answer from the section as it was, and later edits wait to
66apply to the reread section; the session then reports the section's pages as `Changed`.
6167
62## Sessions68## Sessions
6369
crates/notebook/src/lib.rs+129-6
...@@ -281,8 +281,8 @@ impl Replica {...@@ -281,8 +281,8 @@ impl Replica {
281 })281 })
282 }282 }
283283
284 fn lock(&self) -> Result<MutexGuard<'_, Connection>> {284 fn lock(&self) -> Result<Open<'_>> {
285 lock(&self.section.connection)285 lock(&self.section.cache)
286 }286 }
287287
288 /// Hands `request` to the section thread.288 /// Hands `request` to the section thread.
...@@ -494,10 +494,41 @@ impl Drop for Replica {...@@ -494,10 +494,41 @@ impl Drop for Replica {
494 }494 }
495}495}
496496
497fn lock(connection: &Mutex<Connection>) -> Result<MutexGuard<'_, Connection>> {497/// The open cache's connection, for as long as its lock is held.
498 connection498struct Open<'a>(MutexGuard<'a, working::Cache>);
499
500impl std::ops::Deref for Open<'_> {
501 type Target = Connection;
502
503 fn deref(&self) -> &Connection {
504 match &*self.0 {
505 working::Cache::Open(connection) => connection,
506 _ => unreachable!("Only an open cache is lent"),
507 }
508 }
509}
510
511impl std::ops::DerefMut for Open<'_> {
512 fn deref_mut(&mut self) -> &mut Connection {
513 match &mut *self.0 {
514 working::Cache::Open(connection) => connection,
515 _ => unreachable!("Only an open cache is lent"),
516 }
517 }
518}
519
520/// The cache's connection; `WouldBlock` while a failed commit keeps the cache closed.
521fn lock(cache: &Mutex<working::Cache>) -> Result<Open<'_>> {
522 let cache = cache
499 .lock()523 .lock()
500 .map_err(|_| io::Error::other("Cache owner panicked").into())524 .map_err(|_| io::Error::other("Cache owner panicked"))?;
525 match &*cache {
526 working::Cache::Open(_) => Ok(Open(cache)),
527 working::Cache::Closed { .. } => {
528 Err(io::Error::new(io::ErrorKind::WouldBlock, "The cache reopens once it can").into())
529 }
530 working::Cache::Lost(error) => Err(io::Error::other(error.clone()).into()),
531 }
501}532}
502533
503/// Tells the worker a burst of local edits is durable.534/// Tells the worker a burst of local edits is durable.
...@@ -558,9 +589,101 @@ fn signed(value: u64) -> Result<i64> {...@@ -558,9 +589,101 @@ fn signed(value: u64) -> Result<i64> {
558 Ok(i64::try_from(value).map_err(|_| io::Error::from(io::ErrorKind::InvalidInput))?)589 Ok(i64::try_from(value).map_err(|_| io::Error::from(io::ErrorKind::InvalidInput))?)
559}590}
560591
592/// `connect`, busy while anyone owns the cache: SQLite's lock, or the `Owned` a section holds
593/// while a failed commit keeps the cache closed, checked once SQLite's is held, as that
594/// section takes its own before letting SQLite's go.
595fn cache_connection(path: &Path) -> Result<Connection> {
596 let connection = connect(path)?;
597 connection.query_row("SELECT count(*) FROM sqlite_schema", [], |_| Ok(()))?;
598 let owned = identity(path)
599 .is_ok_and(|identity| OWNED.lock().is_ok_and(|owned| owned.contains(&identity)))
600 || std::fs::File::open(owner_lock(path)).is_ok_and(|lock| {
601 matches!(
602 lock.try_lock_shared(),
603 Err(std::fs::TryLockError::WouldBlock)
604 )
605 });
606 if owned {
607 return Err(rusqlite::Error::SqliteFailure(
608 rusqlite::ffi::Error::new(rusqlite::ffi::SQLITE_BUSY),
609 Some("A section holds the cache while it reopens".into()),
610 )
611 .into());
612 }
613 Ok(connection)
614}
615
616/// The file beside a cache whose lock keeps it owned while a failed commit keeps it closed.
617fn owner_lock(cache: &Path) -> PathBuf {
618 let mut lock = cache.as_os_str().to_owned();
619 lock.push("-lock");
620 lock.into()
621}
622
623/// Caches an `Owned` holds in this process, where a lock on a network filesystem (`flock` over
624/// NFS on Linux) need not exclude.
625static OWNED: Mutex<std::collections::BTreeSet<Identity>> =
626 Mutex::new(std::collections::BTreeSet::new());
627
628/// A cache file however its path is spelled: its device and inode, or elsewhere its
629/// canonical path.
630#[cfg(unix)]
631type Identity = (u64, u64);
632#[cfg(not(unix))]
633type Identity = PathBuf;
634
635fn identity(cache: &Path) -> io::Result<Identity> {
636 #[cfg(unix)]
637 {
638 use std::os::unix::fs::MetadataExt;
639 let metadata = std::fs::metadata(cache)?;
640 Ok((metadata.dev(), metadata.ino()))
641 }
642 #[cfg(not(unix))]
643 std::fs::canonicalize(cache)
644}
645
646/// Ownership of a closed cache, which every open honours until it drops: `<cache>-lock`
647/// locked, and the cache listed in `OWNED`.
648struct Owned {
649 cache: PathBuf,
650 identity: Identity,
651 _lock: std::fs::File,
652}
653
654impl Owned {
655 fn take(cache: &Path) -> Result<Self> {
656 let lock = std::fs::File::options()
657 .create(true)
658 .truncate(false)
659 .write(true)
660 .open(owner_lock(cache))?;
661 lock.try_lock().map_err(io::Error::from)?;
662 let identity = identity(cache)?;
663 #[cfg_attr(unix, allow(clippy::clone_on_copy))]
664 (OWNED.lock())
665 .map_err(|_| io::Error::other("Cache owners panicked"))?
666 .insert(identity.clone());
667 Ok(Self {
668 cache: cache.to_owned(),
669 identity,
670 _lock: lock,
671 })
672 }
673}
674
675impl Drop for Owned {
676 fn drop(&mut self) {
677 if let Ok(mut owned) = OWNED.lock() {
678 owned.remove(&self.identity);
679 }
680 let _ = std::fs::remove_file(owner_lock(&self.cache));
681 }
682}
683
561/// Opens a cache without writing to it: exclusive locking, a full sync of every commit684/// Opens a cache without writing to it: exclusive locking, a full sync of every commit
562/// (`F_FULLFSYNC` on macOS, the WAL's syncs included) and foreign keys, each queried back.685/// (`F_FULLFSYNC` on macOS, the WAL's syncs included) and foreign keys, each queried back.
563fn cache_connection(path: &Path) -> Result<Connection> {686fn connect(path: &Path) -> Result<Connection> {
564 #[cfg(target_arch = "wasm32")]687 #[cfg(target_arch = "wasm32")]
565 fs::install_sqlite();688 fs::install_sqlite();
566 let connection = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE)?;689 let connection = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE)?;
crates/notebook/src/location.rs+16-6
...@@ -191,13 +191,23 @@ fn adopt(from: &Path, to: &Path) -> io::Result<()> {...@@ -191,13 +191,23 @@ fn adopt(from: &Path, to: &Path) -> io::Result<()> {
191 Ok(())191 Ok(())
192}192}
193193
194/// Removes a replica folder that holds no replica.194/// Removes a replica folder that holds no replica, with the owner lock a crash may have left
195/// beside a cache moved away since (`crate::Owned`).
195fn tidy(folder: &Path) {196fn tidy(folder: &Path) {
196 let empty = fs::read_dir(folder).is_ok_and(|mut entries| {197 let Ok(entries) = fs::read_dir(folder).and_then(|entries| {
197 entries.all(|entry| entry.is_ok_and(|entry| entry.file_name() == LOCATION))198 entries
198 });199 .map(|entry| Ok(entry?.file_name()))
199 if empty {200 .collect::<io::Result<Vec<_>>>()
200 let _ = fs::remove_file(folder.join(LOCATION));201 }) else {
202 return;
203 };
204 let left = |name: &std::ffi::OsString| {
205 name == LOCATION || name.to_string_lossy().ends_with(".sqlite-lock")
206 };
207 if entries.iter().all(left) {
208 for name in entries {
209 let _ = fs::remove_file(folder.join(name));
210 }
201 let _ = fs::remove_dir(folder);211 let _ = fs::remove_dir(folder);
202 }212 }
203}213}
crates/notebook/src/session.rs+32-11
...@@ -2186,7 +2186,8 @@ pub struct SyncStatus {...@@ -2186,7 +2186,8 @@ pub struct SyncStatus {
2186 pub synced: Option<u64>,2186 pub synced: Option<u64>,
2187 /// Why the last step failed, until one reaches the section file again.2187 /// Why the last step failed, until one reaches the section file again.
2188 pub error: Option<io::Error>,2188 pub error: Option<io::Error>,
2189 /// Edits the section file does not hold yet, uncertain attempts included.2189 /// Edits the section file does not hold yet, uncertain attempts and edits waiting for the
2190 /// cache included; while a failed commit keeps the cache closed, those queued before it.
2190 pub queued: u64,2191 pub queued: u64,
2191}2192}
21922193
...@@ -2237,11 +2238,23 @@ pub(crate) fn sync_status(...@@ -2237,11 +2238,23 @@ pub(crate) fn sync_status(
2237 synced: impl FnOnce() -> Option<u64>,2238 synced: impl FnOnce() -> Option<u64>,
2238 observed: &Mutex<Observed>,2239 observed: &Mutex<Observed>,
2239) -> Result<SyncStatus> {2240) -> Result<SyncStatus> {
2240 let queued: i64 = crate::lock(&replica.section.connection)?.query_row(2241 let waiting = replica
2241 "SELECT count(*) FROM edits",2242 .section
2242 [],2243 .waiting
2243 |row| row.get(0),2244 .load(std::sync::atomic::Ordering::Relaxed) as u64;
2244 )?;2245 let (queued, failed) = match &*(replica.section.cache.lock())
2246 .map_err(|_| io::Error::other("Cache owner panicked"))?
2247 {
2248 crate::working::Cache::Open(connection) => {
2249 let queued: i64 =
2250 connection.query_row("SELECT count(*) FROM edits", [], |row| row.get(0))?;
2251 (crate::unsigned(queued)?, None)
2252 }
2253 crate::working::Cache::Closed { queued, .. } => {
2254 (*queued, Some("The cache reopens once it can".to_owned()))
2255 }
2256 crate::working::Cache::Lost(error) => (0, Some(error.clone())),
2257 };
2245 let observed = observed2258 let observed = observed
2246 .lock()2259 .lock()
2247 .map_err(|_| io::Error::other("Synchronization observer panicked"))?;2260 .map_err(|_| io::Error::other("Synchronization observer panicked"))?;
...@@ -2249,11 +2262,11 @@ pub(crate) fn sync_status(...@@ -2249,11 +2262,11 @@ pub(crate) fn sync_status(
2249 // Read after `observed`: the worker stores the time before it clears the error, so a2262 // Read after `observed`: the worker stores the time before it clears the error, so a
2250 // cleared error never shows beside the time of the step before it.2263 // cleared error never shows beside the time of the step before it.
2251 synced: synced(),2264 synced: synced(),
2252 error: observed2265 error: failed.map(io::Error::other).or_else(|| {
2253 .as_ref()2266 (observed.as_ref().and_then(Option::as_ref))
2254 .and_then(Option::as_ref)2267 .map(|error| io::Error::new(error.kind(), error.to_string()))
2255 .map(|error| io::Error::new(error.kind(), error.to_string())),2268 }),
2256 queued: crate::unsigned(queued)?,2269 queued: queued + waiting,
2257 })2270 })
2258}2271}
22592272
...@@ -2475,6 +2488,14 @@ impl Section {...@@ -2475,6 +2488,14 @@ impl Section {
2475 }2488 }
2476 });2489 });
2477 let observed: Arc<Mutex<Observed>> = Arc::new(Mutex::new(None));2490 let observed: Arc<Mutex<Observed>> = Arc::new(Mutex::new(None));
2491 if let Ok(mut reread) = replica.section.reread.lock() {
2492 let (sender, notify) = (sender.clone(), Arc::clone(&notify));
2493 *reread = Some(Box::new(move |spaces| {
2494 if sender.send(Event::Changed(spaces)).is_ok() {
2495 notify();
2496 }
2497 }));
2498 }
2478 let mut section = Self {2499 let mut section = Self {
2479 file,2500 file,
2480 replica,2501 replica,
crates/notebook/src/session/tests.rs+21
...@@ -854,3 +854,24 @@ fn edits_the_cache_refuses_wait_in_order_and_land_once() {...@@ -854,3 +854,24 @@ fn edits_the_cache_refuses_wait_in_order_and_land_once() {
854 assert_eq!(waited[0].page.color, Some(0x0065_4321));854 assert_eq!(waited[0].page.color, Some(0x0065_4321));
855 assert!(Replica::open(&cache).unwrap().pending().unwrap().is_empty());855 assert!(Replica::open(&cache).unwrap().pending().unwrap().is_empty());
856}856}
857
858#[test]
859fn a_cache_owned_while_closed_opens_for_nobody_else() {
860 let temporary = tempfile::tempdir().unwrap();
861 let cache = temporary.path().join("owned.sqlite");
862 let source = onestore::create_section("Owned.one", "Original", "Fixture").unwrap();
863 drop(Replica::create(&cache, &source).unwrap());
864 let named = PathBuf::from(crate::connect(&cache).unwrap().path().unwrap());
865 let owned = crate::Owned::take(&named).unwrap();
866 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
867 assert!(crate::closed(&cache).is_err_and(|error| error.busy()));
868 drop(owned);
869 assert!(!crate::owner_lock(&named).exists());
870 drop(Replica::open(&cache).unwrap());
871 // Another process holds only the lock beside the cache.
872 let lock = std::fs::File::create(crate::owner_lock(&named)).unwrap();
873 lock.try_lock().unwrap();
874 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
875 drop(lock);
876 drop(Replica::open(&cache).unwrap());
877}
crates/notebook/src/working.rs+467-234
...@@ -15,7 +15,7 @@ use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params};...@@ -15,7 +15,7 @@ use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params};
15use std::{15use std::{
16 collections::{BTreeMap, BTreeSet, VecDeque},16 collections::{BTreeMap, BTreeSet, VecDeque},
17 io,17 io,
18 path::PathBuf,18 path::{Path, PathBuf},
19 sync::{19 sync::{
20 Arc, Mutex, Weak,20 Arc, Mutex, Weak,
21 atomic::{AtomicUsize, Ordering},21 atomic::{AtomicUsize, Ordering},
...@@ -25,6 +25,7 @@ use std::{...@@ -25,6 +25,7 @@ use std::{
2525
26pub(crate) type Reply<T> = Box<dyn FnOnce(Result<T>) + Send>;26pub(crate) type Reply<T> = Box<dyn FnOnce(Result<T>) + Send>;
27pub(crate) type Notice = Box<dyn Fn(&str) + Send>;27pub(crate) type Notice = Box<dyn Fn(&str) + Send>;
28type Changed = Box<dyn Fn(Vec<ExGuid>) + Send>;
2829
29pub(crate) enum Request {30pub(crate) enum Request {
30 Apply(Accepted),31 Apply(Accepted),
...@@ -87,6 +88,34 @@ pub(crate) struct Sealed {...@@ -87,6 +88,34 @@ pub(crate) struct Sealed {
87 pub transaction: Option<Transaction>,88 pub transaction: Option<Transaction>,
88}89}
8990
91/// The cache's connection, or what a failed commit leaves: the cache closed until it reopens
92/// and reads cold whether the commit whose last edit is `last` landed, `ids` naming its edits
93/// still waiting and `queued` the edits queued before it; or lost where it never will reopen.
94pub(crate) enum Cache {
95 Open(Connection),
96 Closed {
97 last: u64,
98 ids: Vec<u64>,
99 queued: u64,
100 owner: Owner,
101 },
102 Lost(String),
103}
104
105/// What keeps everyone else out of a closed cache: the warm connection, unused, until
106/// `crate::Owned` can be taken in its stead.
107pub(crate) enum Owner {
108 Warm(#[allow(dead_code)] Connection),
109 Owned(#[allow(dead_code)] crate::Owned),
110}
111
112/// Whether a failed commit keeps the cache closed.
113fn closed(cache: &Mutex<Cache>) -> bool {
114 cache
115 .lock()
116 .is_ok_and(|cache| matches!(*cache, Cache::Closed { .. }))
117}
118
90impl Request {119impl Request {
91 fn fail(self, kind: io::ErrorKind, message: &str) {120 fn fail(self, kind: io::ErrorKind, message: &str) {
92 let error = || io::Error::new(kind, message.to_owned()).into();121 let error = || io::Error::new(kind, message.to_owned()).into();
...@@ -112,7 +141,10 @@ impl Request {...@@ -112,7 +141,10 @@ impl Request {
112/// keeps answering page reads from the section as it was; the new thread then takes the141/// keeps answering page reads from the section as it was; the new thread then takes the
113/// requests over, so reads never wait for a rebuild.142/// requests over, so reads never wait for a rebuild.
114pub(crate) struct Thread {143pub(crate) struct Thread {
115 pub(crate) connection: Mutex<Connection>,144 pub(crate) cache: Mutex<Cache>,
145 /// Where the cache reopens after a failed commit; none in the browser, where reopening
146 /// through its file system is unverified.
147 path: Option<PathBuf>,
116 /// The sync worker, which a durable burst of edits wakes.148 /// The sync worker, which a durable burst of edits wakes.
117 pub(crate) worker: Mutex<Weak<Signal>>,149 pub(crate) worker: Mutex<Weak<Signal>>,
118 /// Taken when the replica drops, which ends every section thread.150 /// Taken when the replica drops, which ends every section thread.
...@@ -126,7 +158,9 @@ pub(crate) struct Thread {...@@ -126,7 +158,9 @@ pub(crate) struct Thread {
126 /// A password-protected section's key, under which its queue is sealed too.158 /// A password-protected section's key, under which its queue is sealed too.
127 pub(crate) key: Option<Key>,159 pub(crate) key: Option<Key>,
128 /// The edits the cache last refused for now, as `crate::unwritten` counts them.160 /// The edits the cache last refused for now, as `crate::unwritten` counts them.
129 waiting: AtomicUsize,161 pub(crate) waiting: AtomicUsize,
162 /// Told the pages a reread after a refusal may change, which their views reload.
163 pub(crate) reread: Mutex<Option<Changed>>,
130 /// Last, so the cache closes before background synchronization takes it over.164 /// Last, so the cache closes before background synchronization takes it over.
131 pub(crate) released: crate::Released,165 pub(crate) released: crate::Released,
132}166}
...@@ -309,18 +343,20 @@ mod serving {...@@ -309,18 +343,20 @@ mod serving {
309 }343 }
310}344}
311345
312/// The next request; a second passes without one while `retry`.346/// How often the cache is tried again while edits wait for it or it is closed.
347const RETRY: std::time::Duration = std::time::Duration::from_secs(1);
348
349/// The next request, or a timeout once `retry` passes without one.
313#[cfg(not(target_arch = "wasm32"))]350#[cfg(not(target_arch = "wasm32"))]
314async fn receive(351async fn receive(
315 requests: &mpsc::Receiver<Request>,352 requests: &mpsc::Receiver<Request>,
316 retry: bool,353 retry: Option<std::time::Duration>,
317) -> std::result::Result<Request, mpsc::RecvTimeoutError> {354) -> std::result::Result<Request, mpsc::RecvTimeoutError> {
318 if retry {355 match retry {
319 requests.recv_timeout(std::time::Duration::from_secs(1))356 Some(wait) => requests.recv_timeout(wait),
320 } else {357 None => requests
321 requests
322 .recv()358 .recv()
323 .map_err(|_| mpsc::RecvTimeoutError::Disconnected)359 .map_err(|_| mpsc::RecvTimeoutError::Disconnected),
324 }360 }
325}361}
326362
...@@ -329,7 +365,7 @@ async fn receive(...@@ -329,7 +365,7 @@ async fn receive(
329#[cfg(target_arch = "wasm32")]365#[cfg(target_arch = "wasm32")]
330async fn receive(366async fn receive(
331 requests: &mpsc::Receiver<Request>,367 requests: &mpsc::Receiver<Request>,
332 _retry: bool,368 _retry: Option<std::time::Duration>,
333) -> std::result::Result<Request, mpsc::RecvTimeoutError> {369) -> std::result::Result<Request, mpsc::RecvTimeoutError> {
334 std::future::poll_fn(|_| match requests.try_recv() {370 std::future::poll_fn(|_| match requests.try_recv() {
335 Ok(request) => std::task::Poll::Ready(Ok(request)),371 Ok(request) => std::task::Poll::Ready(Ok(request)),
...@@ -349,7 +385,10 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr...@@ -349,7 +385,10 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr
349 #[cfg(target_arch = "wasm32")]385 #[cfg(target_arch = "wasm32")]
350 let (durable_wake, woken) = crate::task::channel();386 let (durable_wake, woken) = crate::task::channel();
351 let shared = Arc::new(Thread {387 let shared = Arc::new(Thread {
352 connection: Mutex::new(connection),388 path: (connection.path())
389 .filter(|_| cfg!(not(target_arch = "wasm32")))
390 .map(PathBuf::from),
391 cache: Mutex::new(Cache::Open(connection)),
353 worker: Mutex::new(Weak::new()),392 worker: Mutex::new(Weak::new()),
354 sender: Mutex::new(Some(sender)),393 sender: Mutex::new(Some(sender)),
355 #[cfg(target_arch = "wasm32")]394 #[cfg(target_arch = "wasm32")]
...@@ -360,6 +399,7 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr...@@ -360,6 +399,7 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr
360 threads: Mutex::new(Vec::new()),399 threads: Mutex::new(Vec::new()),
361 key,400 key,
362 waiting: AtomicUsize::new(0),401 waiting: AtomicUsize::new(0),
402 reread: Mutex::new(None),
363 released: crate::Released::default(),403 released: crate::Released::default(),
364 });404 });
365 #[cfg(target_arch = "wasm32")]405 #[cfg(target_arch = "wasm32")]
...@@ -393,7 +433,7 @@ async fn run(...@@ -393,7 +433,7 @@ async fn run(
393) {433) {
394 loop {434 loop {
395 let arena = Arena::default();435 let arena = Arena::default();
396 let working = match Working::open(&arena, &shared.connection, shared.key.as_ref()) {436 let working = match Working::open(&arena, &shared.cache, shared.key.as_ref()) {
397 Ok(working) => working,437 Ok(working) => working,
398 Err(error) => {438 Err(error) => {
399 if let Some(ready) = ready.take() {439 if let Some(ready) = ready.take() {
...@@ -451,19 +491,17 @@ enum Job {...@@ -451,19 +491,17 @@ enum Job {
451fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) {491fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) {
452 let answer: Box<dyn FnOnce() + Send> = match job {492 let answer: Box<dyn FnOnce() + Send> = match job {
453 Job::Reopen { reply } => Box::new(move || reply(Ok(()))),493 Job::Reopen { reply } => Box::new(move || reply(Ok(()))),
454 Job::Rebase { image, reply } => {494 Job::Rebase { image, reply } => match rebase(&shared.cache, image, shared.key.as_ref()) {
455 match rebase(&shared.connection, image, shared.key.as_ref()) {495 Ok(changed) => Box::new(move || reply(Ok(changed))),
456 Ok(changed) => Box::new(move || reply(Ok(changed))),496 Err(error) => {
457 Err(error) => {497 reply(Err(error));
458 reply(Err(error));498 let _ = signal.send(Request::Resume);
459 let _ = signal.send(Request::Resume);499 return;
460 return;
461 }
462 }500 }
463 }501 },
464 };502 };
465 let arena = Arena::default();503 let arena = Arena::default();
466 let opened = Working::open(&arena, &shared.connection, shared.key.as_ref());504 let opened = Working::open(&arena, &shared.cache, shared.key.as_ref());
467 let (to, from) = mpsc::sync_channel(1);505 let (to, from) = mpsc::sync_channel(1);
468 if signal.send(Request::Handover(to)).is_err() {506 if signal.send(Request::Handover(to)).is_err() {
469 return;507 return;
...@@ -492,40 +530,65 @@ fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) {...@@ -492,40 +530,65 @@ fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) {
492fn inline(shared: &Thread, job: Job) -> bool {530fn inline(shared: &Thread, job: Job) -> bool {
493 match job {531 match job {
494 Job::Reopen { reply } => reply(Ok(())),532 Job::Reopen { reply } => reply(Ok(())),
495 Job::Rebase { image, reply } => {533 Job::Rebase { image, reply } => match rebase(&shared.cache, image, shared.key.as_ref()) {
496 match rebase(&shared.connection, image, shared.key.as_ref()) {534 Ok(changed) => reply(Ok(changed)),
497 Ok(changed) => reply(Ok(changed)),535 Err(error) => {
498 Err(error) => {536 reply(Err(error));
499 reply(Err(error));537 return false;
500 return false;
501 }
502 }538 }
503 }539 },
504 }540 }
505 true541 true
506}542}
507543
508/// Writes again each second the edits a dropped replica left waiting, until none do or their544/// Writes again each second the edits a dropped replica left waiting, until none do and the
509/// cache is gone, keeping the cache open and `Released` from firing meanwhile.545/// cache is open, or the cache is gone; the cache stays owned and `Released` unfired meanwhile.
546/// With `reread`, the closed commit is settled first and the edits applied again to the queue
547/// as it then stands, refusing those that no longer apply, as the section thread would have.
510#[cfg(not(target_arch = "wasm32"))]548#[cfg(not(target_arch = "wasm32"))]
511fn orphan(shared: &Arc<Thread>, unwritten: Vec<Accepted>) {549fn orphan(shared: &Arc<Thread>, mut open: Option<i64>, mut unwritten: Vec<Accepted>, reread: bool) {
512 let shared = Arc::clone(shared);550 let shared = Arc::clone(shared);
513 let path = lock(&shared.connection)
514 .ok()
515 .and_then(|connection| connection.path().map(PathBuf::from));
516 let spawned = std::thread::Builder::new()551 let spawned = std::thread::Builder::new()
517 .name("onestore-unwritten".into())552 .name("onestore-unwritten".into())
518 .spawn(move || {553 .spawn(move || {
519 let arena = Arena::default();
520 let Ok(mut working) = Working::open(&arena, &shared.connection, shared.key.as_ref())
521 else {
522 return;
523 };
524 working.unwritten = unwritten;
525 // A cache deleted, as with its notebook, takes no edits.554 // A cache deleted, as with its notebook, takes no edits.
526 while !working.unwritten.is_empty() && path.as_ref().is_some_and(|path| path.exists()) {555 let kept = || shared.path.as_ref().is_some_and(|path| path.exists());
527 std::thread::sleep(std::time::Duration::from_secs(1));556 let second = || std::thread::sleep(std::time::Duration::from_secs(1));
528 let _ = working.flush(&shared);557 while closed(&shared.cache) && kept() {
558 second();
559 let _ = write(&shared, &mut open, &mut unwritten, !reread);
560 }
561 // An edit that applies part way is refused, and the others apply again to a fresh
562 // reading of the queue.
563 if reread {
564 'reread: loop {
565 let arena = Arena::default();
566 let mut working =
567 match Working::open(&arena, &shared.cache, shared.key.as_ref()) {
568 Ok(working) => working,
569 Err(error) => {
570 for edit in unwritten.drain(..) {
571 (edit.reply)(Err(io::Error::other(error.to_string()).into()));
572 }
573 break;
574 }
575 };
576 let mut edits = std::mem::take(&mut unwritten).into_iter();
577 while let Some(edit) = edits.next() {
578 if working.accept(edit) {
579 unwritten = std::mem::take(&mut working.unwritten);
580 unwritten.extend(edits);
581 continue 'reread;
582 }
583 }
584 open = working.open;
585 unwritten = std::mem::take(&mut working.unwritten);
586 break;
587 }
588 }
589 while !unwritten.is_empty() && kept() {
590 second();
591 let _ = write(&shared, &mut open, &mut unwritten, true);
529 }592 }
530 });593 });
531 if let Err(error) = spawned {594 if let Err(error) = spawned {
...@@ -556,7 +619,7 @@ struct Working<'a> {...@@ -556,7 +619,7 @@ struct Working<'a> {
556}619}
557620
558impl<'a> Working<'a> {621impl<'a> Working<'a> {
559 fn open(arena: &'a Arena, connection: &Mutex<Connection>, key: Option<&Key>) -> Result<Self> {622 fn open(arena: &'a Arena, connection: &Mutex<Cache>, key: Option<&Key>) -> Result<Self> {
560 let (section, open, touched) = replay(arena, &*lock(connection)?, key)?;623 let (section, open, touched) = replay(arena, &*lock(connection)?, key)?;
561 Ok(Self {624 Ok(Self {
562 section,625 section,
...@@ -572,23 +635,53 @@ impl<'a> Working<'a> {...@@ -572,23 +635,53 @@ impl<'a> Working<'a> {
572 requests: &mpsc::Receiver<Request>,635 requests: &mpsc::Receiver<Request>,
573 backlog: &mut VecDeque<Request>,636 backlog: &mut VecDeque<Request>,
574 ) -> Next {637 ) -> Next {
575 let connection = &shared.connection;638 let connection = &shared.cache;
576 // While another thread rebuilds the section, reads answer from this one and every639 // While another thread rebuilds the section, reads answer from this one and every
577 // other request waits for the rebuilt section.640 // other request waits for the rebuilt section.
578 let mut held: Option<VecDeque<Request>> = None;641 let mut held: Option<VecDeque<Request>> = None;
642 // While a reread waits for the closed cache, reads answer from this section and edits
643 // wait for the reread, as one this section refused may underlie them.
644 let mut stale: Option<VecDeque<Request>> = None;
645 let mut attempted = web_time::Instant::now();
579 loop {646 loop {
580 let mut burst = std::mem::take(backlog);647 let mut burst = std::mem::take(backlog);
581 // Only a burst that took an edit, or a retry's tick, writes at its end: reads never.648 // Only a burst that took an edit, or a retry falling due, writes at its end: reads
649 // never.
582 let mut write = false;650 let mut write = false;
651 let retry = !self.unwritten.is_empty() || closed(connection);
583 if burst.is_empty() {652 if burst.is_empty() {
584 match receive(requests, !self.unwritten.is_empty()).await {653 let wait = retry.then(|| RETRY.saturating_sub(attempted.elapsed()));
654 match receive(requests, wait).await {
585 Ok(request) => burst.push_back(request),655 Ok(request) => burst.push_back(request),
586 Err(mpsc::RecvTimeoutError::Timeout) => write = true,656 Err(mpsc::RecvTimeoutError::Timeout) => {}
587 Err(mpsc::RecvTimeoutError::Disconnected) => {657 Err(mpsc::RecvTimeoutError::Disconnected) => {
588 let _ = self.flush(shared);658 let stale_section = stale.is_some();
659 // A refusal now leaves the section holding an edit the queue lacks too.
660 #[cfg_attr(target_arch = "wasm32", allow(unused_variables))]
661 let reread = crate::working::write(
662 shared,
663 &mut self.open,
664 &mut self.unwritten,
665 !stale_section,
666 )
667 .is_err()
668 || stale_section;
669 for request in stale.into_iter().flatten() {
670 match request {
671 Request::Apply(accepted) => self.unwritten.push(accepted),
672 request => {
673 request.fail(io::ErrorKind::Other, "The section thread stopped")
674 }
675 }
676 }
589 #[cfg(not(target_arch = "wasm32"))]677 #[cfg(not(target_arch = "wasm32"))]
590 if !self.unwritten.is_empty() {678 if !self.unwritten.is_empty() || closed(connection) {
591 orphan(shared, std::mem::take(&mut self.unwritten));679 orphan(
680 shared,
681 self.open,
682 std::mem::take(&mut self.unwritten),
683 reread,
684 );
592 }685 }
593 return Next::Stop;686 return Next::Stop;
594 }687 }
...@@ -599,24 +692,11 @@ impl<'a> Working<'a> {...@@ -599,24 +692,11 @@ impl<'a> Working<'a> {
599 let mut later = VecDeque::new();692 let mut later = VecDeque::new();
600 let mut reopen = false;693 let mut reopen = false;
601 while !reopen && let Some(request) = burst.pop_front().or_else(|| later.pop_front()) {694 while !reopen && let Some(request) = burst.pop_front().or_else(|| later.pop_front()) {
695 let Some(request) = self.answer(request) else {
696 continue;
697 };
602 if let Some(waiting) = &mut held {698 if let Some(waiting) = &mut held {
603 match request {699 match request {
604 Request::Page { space, reply } => {
605 reply(self.section.page(space).map_err(Into::into))
606 }
607 Request::Pages { reply } => reply(self.section.pages().map_err(Into::into)),
608 Request::Conflicts { reply } => {
609 reply(self.section.conflicts().map_err(Into::into))
610 }
611 Request::Versions { reply } => {
612 reply(self.section.versions().map_err(Into::into))
613 }
614 Request::Version {
615 space,
616 version,
617 reply,
618 } => reply(self.section.version(space, version).map_err(Into::into)),
619 Request::Moving { space, reply } => reply(self.moving(space)),
620 Request::Handover(to) => {700 Request::Handover(to) => {
621 let mut waiting = held.take().unwrap_or_default();701 let mut waiting = held.take().unwrap_or_default();
622 waiting.extend(burst.drain(..).chain(later.drain(..)));702 waiting.extend(burst.drain(..).chain(later.drain(..)));
...@@ -630,6 +710,25 @@ impl<'a> Working<'a> {...@@ -630,6 +710,25 @@ impl<'a> Working<'a> {
630 }710 }
631 continue;711 continue;
632 }712 }
713 if let Some(waiting) = &mut stale {
714 match request {
715 Request::Apply(
716 accepted @ Accepted {
717 waiting: Some(_), ..
718 },
719 ) => {
720 if let Some(notice) = &accepted.waiting {
721 notice("The cache reopens once it can");
722 }
723 waiting.push_back(Request::Apply(accepted));
724 }
725 request @ Request::Reopen { .. } => waiting.push_back(request),
726 request => {
727 request.fail(io::ErrorKind::WouldBlock, "The cache reopens once it can")
728 }
729 }
730 continue;
731 }
633 if matches!(request, Request::Flush { .. } | Request::Seal { .. })732 if matches!(request, Request::Flush { .. } | Request::Seal { .. })
634 && !burst.is_empty()733 && !burst.is_empty()
635 {734 {
...@@ -641,7 +740,7 @@ impl<'a> Working<'a> {...@@ -641,7 +740,7 @@ impl<'a> Working<'a> {
641 Request::Flush { .. } | Request::Seal { .. } | Request::Rebase { .. }740 Request::Flush { .. } | Request::Seal { .. } | Request::Rebase { .. }
642 ) {741 ) {
643 let failed = self.flush(shared).is_err();742 let failed = self.flush(shared).is_err();
644 if failed || !self.unwritten.is_empty() {743 if failed || !self.unwritten.is_empty() || closed(connection) {
645 // The worker retries later; the edits wait here meanwhile.744 // The worker retries later; the edits wait here meanwhile.
646 request.fail(io::ErrorKind::WouldBlock, "Edits wait to be written");745 request.fail(io::ErrorKind::WouldBlock, "Edits wait to be written");
647 reopen = failed;746 reopen = failed;
...@@ -650,36 +749,17 @@ impl<'a> Working<'a> {...@@ -650,36 +749,17 @@ impl<'a> Working<'a> {
650 }749 }
651 reopen = match request {750 reopen = match request {
652 Request::Apply(accepted) => {751 Request::Apply(accepted) => {
653 write = true;752 // While the cache is closed, an edit nobody waits for is refused before
654 self.accept(accepted)753 // it applies, instead of showing until a reread can drop it.
655 }754 if accepted.waiting.is_none()
656 Request::Page { space, reply } => {755 && let Err(error) = lock(connection).map(drop)
657 reply(self.section.page(space).map_err(Into::into));756 {
658 false757 (accepted.reply)(Err(error));
659 }758 false
660 Request::Pages { reply } => {759 } else {
661 reply(self.section.pages().map_err(Into::into));760 write = true;
662 false761 self.accept(accepted)
663 }762 }
664 Request::Conflicts { reply } => {
665 reply(self.section.conflicts().map_err(Into::into));
666 false
667 }
668 Request::Versions { reply } => {
669 reply(self.section.versions().map_err(Into::into));
670 false
671 }
672 Request::Version {
673 space,
674 version,
675 reply,
676 } => {
677 reply(self.section.version(space, version).map_err(Into::into));
678 false
679 }
680 Request::Moving { space, reply } => {
681 reply(self.moving(space));
682 false
683 }763 }
684 Request::Flush { reply, durable } => {764 Request::Flush { reply, durable } => {
685 #[cfg(target_arch = "wasm32")]765 #[cfg(target_arch = "wasm32")]
...@@ -715,6 +795,12 @@ impl<'a> Working<'a> {...@@ -715,6 +795,12 @@ impl<'a> Working<'a> {
715 }795 }
716 Request::Reopen { reply } => {796 Request::Reopen { reply } => {
717 let _ = self.flush(shared);797 let _ = self.flush(shared);
798 if closed(connection) {
799 stale
800 .get_or_insert_default()
801 .push_back(Request::Reopen { reply });
802 continue;
803 }
718 #[cfg(target_arch = "wasm32")]804 #[cfg(target_arch = "wasm32")]
719 let reopen = inline(shared, Job::Reopen { reply });805 let reopen = inline(shared, Job::Reopen { reply });
720 #[cfg(not(target_arch = "wasm32"))]806 #[cfg(not(target_arch = "wasm32"))]
...@@ -726,20 +812,86 @@ impl<'a> Working<'a> {...@@ -726,20 +812,86 @@ impl<'a> Working<'a> {
726 };812 };
727 reopen813 reopen
728 }814 }
729 Request::Handover(_) | Request::Resume => false,815 Request::Page { .. }
816 | Request::Pages { .. }
817 | Request::Conflicts { .. }
818 | Request::Versions { .. }
819 | Request::Version { .. }
820 | Request::Moving { .. }
821 | Request::Handover(_)
822 | Request::Resume => false,
823 };
824 }
825 // Retries fall due once a second however busy the section is; a closed cache is
826 // tried then alone, not with each edit.
827 if retry {
828 let due = attempted.elapsed() >= RETRY;
829 write = if closed(connection) {
830 due
831 } else {
832 write || due
730 };833 };
731 }834 }
732 if reopen || write && self.flush(shared).is_err() {835 if write {
733 backlog.extend(836 attempted = web_time::Instant::now();
734 self.unwritten837 }
735 .drain(..)838 if let Some(waiting) = &stale {
736 .map(Request::Apply)839 // Only whether the closed commit landed is settled before the reread.
737 .chain(burst.drain(..))840 if write {
738 .chain(later.drain(..)),841 let _ =
739 );842 crate::working::write(shared, &mut self.open, &mut self.unwritten, false);
740 return Next::Reopen;843 }
844 if closed(connection) {
845 let held = waiting
846 .iter()
847 .filter(|request| matches!(request, Request::Apply(_)))
848 .count();
849 shared.wait(self.unwritten.len() + held);
850 continue;
851 }
852 } else if reopen || write && self.flush(shared).is_err() {
853 if closed(connection) {
854 stale = Some(VecDeque::new());
855 backlog.extend(burst.drain(..).chain(later.drain(..)));
856 continue;
857 }
858 } else {
859 continue;
860 }
861 backlog.extend(
862 self.unwritten
863 .drain(..)
864 .map(Request::Apply)
865 .chain(stale.into_iter().flatten())
866 .chain(burst.drain(..))
867 .chain(later.drain(..)),
868 );
869 // What the host asks next answers from the reread section.
870 if let (Ok(pages), Ok(reread)) = (self.section.pages(), shared.reread.lock())
871 && let Some(reread) = reread.as_ref()
872 {
873 reread(pages.into_iter().map(|(space, ..)| space).collect());
741 }874 }
875 return Next::Reopen;
876 }
877 }
878
879 /// Answers `request` where it reads the section; any other comes back.
880 fn answer(&mut self, request: Request) -> Option<Request> {
881 match request {
882 Request::Page { space, reply } => reply(self.section.page(space).map_err(Into::into)),
883 Request::Pages { reply } => reply(self.section.pages().map_err(Into::into)),
884 Request::Conflicts { reply } => reply(self.section.conflicts().map_err(Into::into)),
885 Request::Versions { reply } => reply(self.section.versions().map_err(Into::into)),
886 Request::Version {
887 space,
888 version,
889 reply,
890 } => reply(self.section.version(space, version).map_err(Into::into)),
891 Request::Moving { space, reply } => reply(self.moving(space)),
892 request => return Some(request),
742 }893 }
894 None
743 }895 }
744896
745 /// Starts a thread that runs `job` and rereads the section; false when the replica is897 /// Starts a thread that runs `job` and rereads the section; false when the replica is
...@@ -807,109 +959,12 @@ impl<'a> Working<'a> {...@@ -807,109 +959,12 @@ impl<'a> Working<'a> {
807 }959 }
808 }960 }
809961
810 /// Writes the unwritten edits in one transaction, then answers their senders. A write
811 /// that may succeed later keeps the edits with a notice, telling the first; one that never
812 /// will refuses the edit it failed on. An error means the section holds an edit the queue
813 /// never will, or the cache closed, and must be reread.
814 fn flush(&mut self, shared: &Arc<Thread>) -> Result<()> {962 fn flush(&mut self, shared: &Arc<Thread>) -> Result<()> {
815 if self.unwritten.is_empty() {963 write(shared, &mut self.open, &mut self.unwritten, true)
816 return Ok(());
817 }
818 let mut connection = lock(&shared.connection)?;
819 let mut ids = Vec::new();
820 let committed = connection
821 .transaction_with_behavior(TransactionBehavior::Immediate)
822 .map_err(crate::Error::from)
823 .and_then(|transaction| {
824 let batch = match self.open {
825 Some(batch) => batch,
826 None => {
827 transaction.execute("INSERT INTO batches DEFAULT VALUES", [])?;
828 transaction.last_insert_rowid()
829 }
830 };
831 for edit in &self.unwritten {
832 ids.push(queue::insert(
833 &transaction,
834 shared.key.as_ref(),
835 None,
836 batch,
837 &edit.author,
838 &edit.edit,
839 )?);
840 }
841 Ok((batch, transaction.commit().err()))
842 });
843 // The error, and the edit it refuses when it would fail again.
844 let written = match committed {
845 Ok((batch, None)) => Ok(batch),
846 Ok((batch, Some(error))) => {
847 if landed(&mut connection, ids.last().copied().unwrap_or_default())? {
848 Ok(batch)
849 } else {
850 Err((error.into(), None))
851 }
852 }
853 Err(error) => {
854 let refused = (!transient(&error)).then_some(ids.len());
855 Err((error, refused))
856 }
857 };
858 drop(connection);
859 let batch = match written {
860 Ok(batch) => batch,
861 Err((error, refused)) => {
862 let message = error.to_string();
863 let before = self.unwritten.len();
864 let mut at = 0;
865 for edit in self.unwritten.extract_if(.., |edit| {
866 at += 1;
867 edit.waiting.is_none() || refused == Some(at - 1)
868 }) {
869 (edit.reply)(Err(io::Error::other(message.clone()).into()));
870 }
871 shared.wait(self.unwritten.len());
872 if let Some(notice) = self
873 .unwritten
874 .first()
875 .and_then(|edit| edit.waiting.as_ref())
876 {
877 notice(&message);
878 }
879 return if self.unwritten.len() == before {
880 Ok(())
881 } else {
882 Err(error)
883 };
884 }
885 };
886 self.open = Some(batch);
887 #[cfg(target_arch = "wasm32")]
888 shared.receipt(
889 self.unwritten
890 .drain(..)
891 .zip(ids)
892 .map(|(edit, id)| {
893 let reply: Reply<()> =
894 Box::new(move |result| (edit.reply)(result.map(|()| id)));
895 reply
896 })
897 .collect(),
898 true,
899 );
900 #[cfg(not(target_arch = "wasm32"))]
901 {
902 for (edit, id) in self.unwritten.drain(..).zip(ids) {
903 (edit.reply)(Ok(id));
904 }
905 crate::edited(&shared.worker);
906 }
907 shared.wait(0);
908 Ok(())
909 }964 }
910965
911 /// Seals the open batch unless a sealed batch still waits for publication.966 /// Seals the open batch unless a sealed batch still waits for publication.
912 fn seal(&mut self, connection: &Mutex<Connection>) -> Result<Option<Sealed>> {967 fn seal(&mut self, connection: &Mutex<Cache>) -> Result<Option<Sealed>> {
913 let Some(batch) = self.open else {968 let Some(batch) = self.open else {
914 return Ok(None);969 return Ok(None);
915 };970 };
...@@ -954,7 +1009,7 @@ impl<'a> Working<'a> {...@@ -954,7 +1009,7 @@ impl<'a> Working<'a> {
954/// version the remote could not take gains a conflict page holding it, queued as a new edit.1009/// version the remote could not take gains a conflict page holding it, queued as a new edit.
955/// Returns the pages the remote changed.1010/// Returns the pages the remote changed.
956fn rebase(1011fn rebase(
957 connection: &Mutex<Connection>,1012 connection: &Mutex<Cache>,
958 image: Option<Vec<u8>>,1013 image: Option<Vec<u8>>,
959 key: Option<&Key>,1014 key: Option<&Key>,
960) -> Result<Vec<ExGuid>> {1015) -> Result<Vec<ExGuid>> {
...@@ -1186,28 +1241,206 @@ fn transient(error: &crate::Error) -> bool {...@@ -1186,28 +1241,206 @@ fn transient(error: &crate::Error) -> bool {
1186 if matches!(error.code, SystemIoFailure | DiskFull | DatabaseBusy | DatabaseLocked | CannotOpen))1241 if matches!(error.code, SystemIoFailure | DiskFull | DatabaseBusy | DatabaseLocked | CannotOpen))
1187}1242}
11881243
1189/// Whether the commit whose last edit is `last` reached the cache after it failed, as a cold1244/// Writes `unwritten` in one transaction after the open batch `open`, then answers their
1190/// open reads it, a commit landing whole: the warm connection may not know of frames its WAL1245/// senders. A write that may succeed later keeps the edits with a notice; one that never will
1191/// holds, and must never write again. When the cache cannot be reopened, an inert connection1246/// refuses the edit it failed on, and an edit nobody waits for is refused at once. A failed
1192/// stays, failing every statement.1247/// commit leaves the cache closed, still owned, until it reopens to decide whether the commit
1193fn landed(connection: &mut Connection, last: u64) -> Result<bool> {1248/// landed; a lost cache refuses every edit. An error means the section holds an edit the
1194 let path = connection.path().map(PathBuf::from);1249/// queue never will, or the cache is lost, and must be reread.
1195 let inert = Connection::open_in_memory()?;1250fn write(
1196 inert.execute_batch("PRAGMA query_only=1")?;1251 shared: &Arc<Thread>,
1197 std::mem::replace(connection, inert)1252 open: &mut Option<i64>,
1198 .close()1253 unwritten: &mut Vec<Accepted>,
1199 .map_err(|(_, error)| error)?;1254 commit: bool,
1255) -> Result<()> {
1256 let mut cache = (shared.cache.lock()).map_err(|_| io::Error::other("Cache owner panicked"))?;
1257 if unwritten.is_empty() && !matches!(*cache, Cache::Closed { .. }) {
1258 return Ok(());
1259 }
1260 let path = shared.path.as_deref();
1261 // The ids the cache holds for the first of the edits.
1262 let mut ids = Vec::new();
1263 // The error, and the edit it refuses when it would fail again.
1264 let failed = (|| {
1265 ids = reopen(&mut cache, path, open).map_err(|error| (error, None))?;
1266 if !commit || ids.len() == unwritten.len() {
1267 return Ok(());
1268 }
1269 let Cache::Open(connection) = &mut *cache else {
1270 unreachable!("A cache that does not reopen fails");
1271 };
1272 let mut inserted = Vec::new();
1273 let mut queued = 0;
1274 let committed = connection
1275 .transaction_with_behavior(TransactionBehavior::Immediate)
1276 .map_err(crate::Error::from)
1277 .and_then(|transaction| {
1278 queued = transaction
1279 .query_row("SELECT count(*) FROM edits", [], |row| row.get::<_, i64>(0))?;
1280 let batch = match *open {
1281 Some(batch) => batch,
1282 None => {
1283 transaction.execute("INSERT INTO batches DEFAULT VALUES", [])?;
1284 transaction.last_insert_rowid()
1285 }
1286 };
1287 for edit in &unwritten[ids.len()..] {
1288 inserted.push(queue::insert(
1289 &transaction,
1290 shared.key.as_ref(),
1291 None,
1292 batch,
1293 &edit.author,
1294 &edit.edit,
1295 )?);
1296 }
1297 Ok((batch, transaction.commit().err()))
1298 });
1299 match committed {
1300 Ok((batch, None)) => *open = Some(batch),
1301 Ok((_, Some(error))) => {
1302 // The warm connection may not know of frames its WAL holds, and must never
1303 // write again; it keeps the cache owned until the owner lock can.
1304 if let Cache::Open(warm) =
1305 std::mem::replace(&mut *cache, Cache::Lost(String::new()))
1306 {
1307 *cache = Cache::Closed {
1308 last: inserted[inserted.len() - 1],
1309 ids: inserted,
1310 queued: queued.unsigned_abs(),
1311 owner: Owner::Warm(warm),
1312 };
1313 }
1314 inserted = reopen(&mut cache, path, open).map_err(|error| (error, None))?;
1315 if inserted.is_empty() {
1316 return Err((error.into(), None));
1317 }
1318 }
1319 Err(error) => {
1320 let refused = (!transient(&error)).then_some(inserted.len());
1321 return Err((error, refused));
1322 }
1323 }
1324 ids.extend(inserted);
1325 Ok(())
1326 })();
1327 let written: Vec<_> = unwritten.drain(..ids.len()).zip(ids).collect();
1328 let refused: Vec<_> = match &failed {
1329 Ok(()) => Vec::new(),
1330 Err((_, failing)) => {
1331 let lost = matches!(*cache, Cache::Lost(_));
1332 let (mut at, mut kept) = (0, 0);
1333 (unwritten.extract_if(.., |edit| {
1334 at += 1;
1335 let refuse = lost || edit.waiting.is_none() || *failing == Some(at - 1);
1336 if !refuse {
1337 kept += 1;
1338 } else if let Cache::Closed { ids, .. } = &mut *cache
1339 && kept < ids.len()
1340 {
1341 // Whether its commit landed, it is answered now; a reread shows which.
1342 ids.remove(kept);
1343 }
1344 refuse
1345 }))
1346 .collect()
1347 }
1348 };
1349 drop(cache);
1350 // Counted before anyone hears, as a host hearing may ask.
1351 shared.wait(unwritten.len());
1352 if !written.is_empty() {
1353 #[cfg(target_arch = "wasm32")]
1354 shared.receipt(
1355 written
1356 .into_iter()
1357 .map(|(edit, id)| {
1358 let reply: Reply<()> =
1359 Box::new(move |result| (edit.reply)(result.map(|()| id)));
1360 reply
1361 })
1362 .collect(),
1363 true,
1364 );
1365 #[cfg(not(target_arch = "wasm32"))]
1366 {
1367 for (edit, id) in written {
1368 (edit.reply)(Ok(id));
1369 }
1370 crate::edited(&shared.worker);
1371 }
1372 }
1373 let Err((error, _)) = failed else {
1374 return Ok(());
1375 };
1376 let message = error.to_string();
1377 let kept = refused.is_empty();
1378 for edit in refused {
1379 (edit.reply)(Err(io::Error::other(message.clone()).into()));
1380 }
1381 if let Some(notice) = unwritten.iter().find_map(|edit| edit.waiting.as_ref()) {
1382 notice(&message);
1383 }
1384 if kept { Ok(()) } else { Err(error) }
1385}
1386
1387/// Reopens the cache a failed commit closed and reads cold whether the commit landed, as one
1388/// lands whole: the ids of its edits still waiting when it did, its batch becoming `open`;
1389/// none where it did not or the cache was open. The closed cache stays owned meanwhile, and
1390/// is lost once the file is gone or no longer a database.
1391fn reopen(cache: &mut Cache, path: Option<&Path>, open: &mut Option<i64>) -> Result<Vec<u64>> {
1392 let (last, ids, owner) = match cache {
1393 Cache::Open(_) => return Ok(Vec::new()),
1394 Cache::Closed {
1395 last, ids, owner, ..
1396 } => (*last, ids, owner),
1397 Cache::Lost(error) => return Err(io::Error::other(error.clone()).into()),
1398 };
1200 // Reopening through the browser's file system is unverified: it stays closed.1399 // Reopening through the browser's file system is unverified: it stays closed.
1201 let path = path1400 let Some(path) = path else {
1202 .filter(|_| cfg!(not(target_arch = "wasm32")))1401 let error = "The cache could not confirm a write";
1203 .ok_or_else(|| io::Error::other("The cache could not confirm a write"))?;1402 *cache = Cache::Lost(error.to_owned());
1204 let cold = crate::cache_connection(&path)?;1403 return Err(io::Error::other(error).into());
1205 crate::write_ahead(&cold)?;1404 };
1206 let found: bool = cold.query_row(1405 let reopened = (|| -> Result<(Connection, Option<i64>)> {
1207 "SELECT EXISTS(SELECT 1 FROM edits WHERE id=?1)",1406 if let Owner::Warm(_) = owner {
1208 [crate::signed(last)?],1407 *owner = Owner::Owned(crate::Owned::take(path)?);
1209 |row| row.get(0),1408 }
1210 )?;1409 let cold = crate::connect(path)?;
1211 *connection = cold;1410 crate::write_ahead(&cold)?;
1212 Ok(found)1411 if cold.is_readonly(rusqlite::MAIN_DB)? {
1412 return Err(io::Error::from(io::ErrorKind::ReadOnlyFilesystem).into());
1413 }
1414 let batch = cold
1415 .query_row(
1416 "SELECT batch FROM edits WHERE id=?1",
1417 [crate::signed(last)?],
1418 |row| row.get(0),
1419 )
1420 .optional()?;
1421 Ok((cold, batch))
1422 })();
1423 match reopened {
1424 Ok((cold, batch)) => {
1425 let landed = match batch {
1426 Some(batch) => {
1427 *open = Some(batch);
1428 std::mem::take(ids)
1429 }
1430 None => Vec::new(),
1431 };
1432 *cache = Cache::Open(cold);
1433 Ok(landed)
1434 }
1435 Err(error) => {
1436 use rusqlite::ErrorCode::*;
1437 let lost = !path.exists()
1438 || matches!(&error, crate::Error::Database(rusqlite::Error::SqliteFailure(error, _))
1439 if matches!(error.code, NotADatabase | DatabaseCorrupt));
1440 if lost {
1441 *cache = Cache::Lost(error.to_string());
1442 }
1443 Err(error)
1444 }
1445 }
1213}1446}
crates/notebook/tests/commit_fault.rs+546-55
...@@ -1,20 +1,47 @@...@@ -1,20 +1,47 @@
1//! Commits whose WAL write or sync fails, through a shim over the platform's VFS. Its own1//! Commits whose WAL write or sync fails, and caches that will not reopen after one, through a
2//! binary, as the shim becomes the process's default VFS.2//! shim over the platform's VFS. Its own binary, as the shim becomes the process's default VFS.
3#![cfg(unix)]3#![cfg(unix)]
44
5use notebook::{5use notebook::{
6 Replica,6 EditStatus, Error, Replica,
7 session::{Event, Section},7 session::{Event, Section, SyncState},
8};
9use onestore::{
10 PageCreation,
11 op::{Edit, Op, PageOp, SectionOp},
12};
13use std::{
14 io,
15 path::{Path, PathBuf},
16 sync::{MutexGuard, PoisonError, atomic::Ordering, mpsc},
17 time::{Duration, Instant},
8};18};
9use onestore::op::{Edit, Op, PageOp};
10use std::{path::Path, sync::atomic::Ordering, sync::mpsc, time::Duration};
1119
12#[path = "support/vfs.rs"]20#[path = "support/vfs.rs"]
13mod vfs;21mod vfs;
14use vfs::{22use vfs::{
15 ARMED, KEEP_WAL, OPEN, SYNC_AFTER, SYNC_BEFORE, WAL_WRITES_AFTER_FAULT, WRITE_COMMIT, install,23 ARMED, EXCLUSIVE, FAILING_OPENS, KEEP_WAL, SYNC_AFTER, SYNC_BEFORE, WAL_WRITES_AFTER_FAULT,
24 WRITE_COMMIT, install,
16};25};
1726
27/// Holds the shim for one test, disarmed of what a failed test left.
28fn exclusive() -> MutexGuard<'static, ()> {
29 install();
30 let exclusive = EXCLUSIVE.lock().unwrap_or_else(PoisonError::into_inner);
31 ARMED.store(0, Ordering::SeqCst);
32 FAILING_OPENS.store(0, Ordering::SeqCst);
33 exclusive
34}
35
36/// A section file in `root` and its cache, which the returned path names.
37fn seed(root: &Path) -> PathBuf {
38 let source = onestore::create_section("Fault.one", "Original", "Fixture").unwrap();
39 std::fs::write(root.join("Fault.one"), &source).unwrap();
40 let cache = root.join("replica.sqlite");
41 drop(Replica::create(&cache, &source).unwrap());
42 cache
43}
44
18fn colour(space: onestore::ExGuid, colour: u32) -> Edit {45fn colour(space: onestore::ExGuid, colour: u32) -> Edit {
19 Edit {46 Edit {
20 at: 133_000_000_000_000_000 + u64::from(colour),47 at: 133_000_000_000_000_000 + u64::from(colour),
...@@ -25,17 +52,49 @@ fn colour(space: onestore::ExGuid, colour: u32) -> Edit {...@@ -25,17 +52,49 @@ fn colour(space: onestore::ExGuid, colour: u32) -> Edit {
25 }52 }
26}53}
2754
55/// The queued edits a cold open of `cache` reads, with the colour each gives the page.
56fn colours(cache: &Path) -> Vec<(u64, u32)> {
57 Replica::open(cache)
58 .unwrap()
59 .pending()
60 .unwrap()
61 .iter()
62 .map(|edit| match &edit.edit.ops[..] {
63 [
64 Op::Page {
65 op: PageOp::Color(Some(colour)),
66 ..
67 },
68 ] => (edit.id, *colour),
69 ops => panic!("{ops:?}"),
70 })
71 .collect()
72}
73
74fn until(section: &Section, seen: impl Fn(&Event) -> bool) {
75 let deadline = Instant::now() + Duration::from_secs(10);
76 while !section.events().iter().any(&seen) {
77 assert!(Instant::now() < deadline);
78 std::thread::sleep(Duration::from_millis(10));
79 }
80}
81
82fn would_block<T>(result: Result<T, Error>) -> bool {
83 matches!(result, Err(Error::Io(error)) if error.kind() == io::ErrorKind::WouldBlock)
84}
85
28/// What the section answered for the edit committed under `fault` and the one after it,86/// What the section answered for the edit committed under `fault` and the one after it,
29/// whether a failed write was reported, and the queue as the section saw it.87/// whether a failed write was reported, and the queue as the section saw it.
30struct Outcome {88struct Outcome {
31 replies: [Result<u64, String>; 2],89 replies: [Result<u64, String>; 2],
32 failed: bool,90 failed: bool,
33 pending: Option<Vec<u64>>,91 pending: Vec<u64>,
34}92}
3593
36/// Seeds the WAL with one edit, so the armed commit's sync is the only one it makes, then94/// Seeds the WAL with one edit, so the armed commit's sync is the only one it makes, then
37/// commits under `fault` and once more after it; with `crash`, leaves the cache open.95/// commits under `fault`, the cache failing to reopen `opens` times, and once more after it;
38fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {96/// with `crash`, leaves the cache open.
97fn scenario(root: &Path, fault: u8, opens: usize, crash: bool) -> Outcome {
39 install();98 install();
40 let replica = Replica::open(root.join("replica.sqlite")).unwrap();99 let replica = Replica::open(root.join("replica.sqlite")).unwrap();
41 let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap();100 let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap();
...@@ -54,9 +113,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {...@@ -54,9 +113,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
54 };113 };
55 apply(1).unwrap();114 apply(1).unwrap();
56 WAL_WRITES_AFTER_FAULT.store(0, Ordering::SeqCst);115 WAL_WRITES_AFTER_FAULT.store(0, Ordering::SeqCst);
116 FAILING_OPENS.store(opens, Ordering::SeqCst);
57 ARMED.store(fault, Ordering::SeqCst);117 ARMED.store(fault, Ordering::SeqCst);
58 let first = apply(2);118 let first = apply(2);
59 assert_eq!(ARMED.load(Ordering::SeqCst), 0, "fault {fault} never fired");119 assert_eq!(ARMED.load(Ordering::SeqCst), 0, "fault {fault} never fired");
120 assert_eq!(
121 FAILING_OPENS.load(Ordering::SeqCst),
122 0,
123 "the cache never reopened"
124 );
60 assert_eq!(125 assert_eq!(
61 WAL_WRITES_AFTER_FAULT.load(Ordering::SeqCst),126 WAL_WRITES_AFTER_FAULT.load(Ordering::SeqCst),
62 0,127 0,
...@@ -69,8 +134,10 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {...@@ -69,8 +134,10 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
69 .any(|event| matches!(event, Event::Failed(_)));134 .any(|event| matches!(event, Event::Failed(_)));
70 let pending = section135 let pending = section
71 .pending()136 .pending()
72 .ok()137 .unwrap()
73 .map(|pending| pending.iter().map(|edit| edit.id).collect());138 .iter()
139 .map(|edit| edit.id)
140 .collect();
74 if crash {141 if crash {
75 std::mem::forget(section);142 std::mem::forget(section);
76 } else {143 } else {
...@@ -85,14 +152,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {...@@ -85,14 +152,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome {
85152
86#[test]153#[test]
87fn commit_fault_child() {154fn commit_fault_child() {
88 let (Some(root), Some(fault)) = (155 let (Some(root), Some(fault), Some(opens)) = (
89 std::env::var_os("COMMIT_FAULT_CHILD"),156 std::env::var_os("COMMIT_FAULT_CHILD"),
90 std::env::var("COMMIT_FAULT").ok(),157 std::env::var("COMMIT_FAULT").ok(),
158 std::env::var("COMMIT_FAULT_OPENS").ok(),
91 ) else {159 ) else {
92 return;160 return;
93 };161 };
94 let root = Path::new(&root);162 let root = Path::new(&root);
95 let outcome = scenario(root, fault.parse().unwrap(), true);163 let outcome = scenario(root, fault.parse().unwrap(), opens.parse().unwrap(), true);
96 let ids = |result: &Result<u64, String>| result.as_ref().map_or(0, |id| *id);164 let ids = |result: &Result<u64, String>| result.as_ref().map_or(0, |id| *id);
97 std::fs::write(165 std::fs::write(
98 root.join("outcome"),166 root.join("outcome"),
...@@ -110,68 +178,491 @@ fn commit_fault_child() {...@@ -110,68 +178,491 @@ fn commit_fault_child() {
110178
111#[test]179#[test]
112fn a_failed_commit_is_decided_by_a_cold_read_and_never_replayed_twice() {180fn a_failed_commit_is_decided_by_a_cold_read_and_never_replayed_twice() {
113 install();181 let _exclusive = exclusive();
114 // (fault, whether the edit reached the cache, whether the cache stays closed)182 // (fault, whether the edit reached the cache, how many reopens fail)
115 for (fault, landed, closed) in [183 for (fault, landed, opens) in [
116 (SYNC_BEFORE, false, false),184 (SYNC_BEFORE, false, 0),
117 (SYNC_AFTER, false, false),185 (SYNC_AFTER, false, 0),
118 (WRITE_COMMIT, false, false),186 (WRITE_COMMIT, false, 0),
119 (SYNC_AFTER | KEEP_WAL, true, false),187 (SYNC_AFTER | KEEP_WAL, true, 0),
120 (SYNC_BEFORE | OPEN, false, true),188 (SYNC_BEFORE, false, 2),
189 (SYNC_AFTER | KEEP_WAL, true, 2),
121 ] {190 ] {
122 for crash in [false, true] {191 for crash in [false, true] {
123 let directory = tempfile::tempdir().unwrap();192 let directory = tempfile::tempdir().unwrap();
124 let root = directory.path();193 let root = directory.path();
125 let source = onestore::create_section("Fault.one", "Original", "Fixture").unwrap();194 let cache = seed(root);
126 std::fs::write(root.join("Fault.one"), &source).unwrap();
127 drop(Replica::create(root.join("replica.sqlite"), &source).unwrap());
128 let (first, second, failed) = if crash {195 let (first, second, failed) = if crash {
129 let output = std::process::Command::new(std::env::current_exe().unwrap())196 let output = std::process::Command::new(std::env::current_exe().unwrap())
130 .args(["--exact", "commit_fault_child", "--nocapture"])197 .args(["--exact", "commit_fault_child", "--nocapture"])
131 .env("COMMIT_FAULT_CHILD", root)198 .env("COMMIT_FAULT_CHILD", root)
132 .env("COMMIT_FAULT", fault.to_string())199 .env("COMMIT_FAULT", fault.to_string())
200 .env("COMMIT_FAULT_OPENS", opens.to_string())
133 .output()201 .output()
134 .unwrap();202 .unwrap();
135 assert!(output.status.success(), "{output:?}");203 assert!(output.status.success(), "{output:?}");
136 let outcome = std::fs::read_to_string(root.join("outcome")).unwrap();204 let outcome = std::fs::read_to_string(root.join("outcome")).unwrap();
137 let mut fields = outcome.split(' ');205 let mut fields = outcome.split(' ');
138 let mut id = || Some(fields.next()?.parse::<u64>().unwrap()).filter(|id| *id > 0);206 let mut id = || fields.next().unwrap().parse::<u64>().unwrap();
139 let (first, second) = (id(), id());207 let (first, second) = (id(), id());
140 (first, second, fields.next() == Some("true"))208 (first, second, fields.next() == Some("true"))
141 } else {209 } else {
142 let outcome = scenario(root, fault, false);210 let outcome = scenario(root, fault, opens, false);
143 let [first, second] = outcome.replies;211 let ids = outcome.replies.map(Result::unwrap);
144 if !closed {212 assert_eq!(outcome.pending[1..], ids);
145 let ids = [first.clone().unwrap(), second.clone().unwrap()];213 (ids[0], ids[1], outcome.failed)
146 assert_eq!(&outcome.pending.unwrap()[1..], ids);
147 } else {
148 assert!(outcome.pending.is_none());
149 }
150 (first.ok(), second.ok(), outcome.failed)
151 };214 };
152 assert_eq!(failed, !landed, "fault {fault}");215 // An edit waiting for the cache to reopen is reported, as one that did not land.
153 let cold = Replica::open(root.join("replica.sqlite")).unwrap();216 assert_eq!(failed, !landed || opens > 0, "fault {fault}, {opens} opens");
154 let colours: Vec<_> = cold217 assert_eq!(
155 .pending()218 colours(&cache)[1..],
219 [(first, 2), (second, 3)],
220 "fault {fault}, {opens} opens"
221 );
222 }
223 }
224}
225
226#[test]
227fn edits_wait_while_the_cache_cannot_reopen_and_land_once() {
228 let _exclusive = exclusive();
229 for close in [false, true] {
230 let directory = tempfile::tempdir().unwrap();
231 let root = directory.path();
232 let cache = seed(root);
233 let section = Section::resume(
234 root.join("Fault.one"),
235 Replica::open(&cache).unwrap(),
236 true,
237 || {},
238 )
239 .unwrap();
240 let space = section.pages().unwrap()[0].0;
241 let (replied, replies) = mpsc::channel();
242 let apply = |value| {
243 let replied = replied.clone();
244 section
245 .apply_with("Fixture", colour(space, value), move |result| {
246 replied
247 .send(result.map_err(|error| error.to_string()))
248 .unwrap();
249 })
250 .unwrap();
251 };
252 let reply = || {
253 replies
254 .recv_timeout(Duration::from_secs(10))
156 .unwrap()255 .unwrap()
157 .iter()256 .unwrap()
158 .map(|edit| match &edit.edit.ops[..] {257 };
159 [258 apply(1);
160 Op::Page {259 let seeded = reply();
161 op: PageOp::Color(Some(colour)),260 FAILING_OPENS.store(1000, Ordering::SeqCst);
162 ..261 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
163 },262 apply(2);
164 ] => (edit.id, *colour),263 until(&section, |event| matches!(event, Event::Failed(_)));
165 ops => panic!("{ops:?}"),264 // The cache is tried again once a second.
265 let left = FAILING_OPENS.load(Ordering::SeqCst);
266 std::thread::sleep(Duration::from_secs(3));
267 let tries = left - FAILING_OPENS.load(Ordering::SeqCst);
268 // Never faster than once a second; as slow as the machine is busy.
269 assert!((1..=4).contains(&tries), "{tries} tries in 3 s");
270 apply(3);
271 // A caller blocking on its edit learns at once, and the waiting edits stay.
272 assert!(would_block(
273 section.replica().apply("Fixture", colour(space, 4))
274 ));
275 // A sync step meets the closed cache and the worker carries on.
276 section.wake();
277 until(
278 &section,
279 |event| matches!(event, Event::Failed(error) if error == "The cache reopens once it can"),
280 );
281 // Each second the cache is tried again, and the edits still wait.
282 until(&section, |event| matches!(event, Event::Failed(_)));
283 assert!(replies.try_recv().is_err());
284 assert_eq!(section.page(space).unwrap().color, Some(3));
285 assert!(would_block(section.written()));
286 assert!(would_block(section.pending()));
287 assert_eq!(notebook::unwritten(), 2);
288 let status = section.sync_status().unwrap();
289 // The seed, queued before the commit that failed, and the two waiting.
290 assert_eq!((status.state(), status.queued), (SyncState::Failed, 3));
291 // Nobody else takes the cache meanwhile, nor once the section closes.
292 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
293 let open = if close {
294 section.close().unwrap();
295 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
296 None
297 } else {
298 Some(section)
299 };
300 FAILING_OPENS.store(0, Ordering::SeqCst);
301 let ids = [reply(), reply()];
302 if let Some(section) = open {
303 section.wake();
304 until(&section, |event| {
305 matches!(event, Event::Attempt { id, status: EditStatus::Published { .. } }
306 if *id == ids[1])
307 });
308 section.close().unwrap();
309 let arena = onestore::Arena::default();
310 let stored =
311 onestore::Section::open(&arena, std::fs::read(root.join("Fault.one")).unwrap())
312 .unwrap();
313 assert_eq!(stored.page(space).unwrap().color, Some(3));
314 assert!(colours(&cache).is_empty());
315 } else {
316 // Its cache is released once the edits are written.
317 let deadline = Instant::now() + Duration::from_secs(10);
318 while Replica::open(&cache).is_err() {
319 assert!(Instant::now() < deadline);
320 std::thread::sleep(Duration::from_millis(10));
321 }
322 assert_eq!(colours(&cache), [(seeded, 1), (ids[0], 2), (ids[1], 3)]);
323 }
324 assert_eq!(notebook::unwritten(), 0);
325 }
326}
327
328#[test]
329fn a_lost_cache_refuses_the_edits_that_wait() {
330 let _exclusive = exclusive();
331 for corrupt in [false, true] {
332 let directory = tempfile::tempdir().unwrap();
333 let root = directory.path();
334 let cache = seed(root);
335 let section = Section::resume(
336 root.join("Fault.one"),
337 Replica::open(&cache).unwrap(),
338 true,
339 || {},
340 )
341 .unwrap();
342 let space = section.pages().unwrap()[0].0;
343 let (replied, replies) = mpsc::channel();
344 let apply = |value| {
345 let replied = replied.clone();
346 section
347 .apply_with("Fixture", colour(space, value), move |result| {
348 replied
349 .send(result.map_err(|error| error.to_string()))
350 .unwrap();
166 })351 })
167 .collect();352 .unwrap();
168 if closed {353 };
169 assert_eq!((first, second), (None, None));354 let reply = || replies.recv_timeout(Duration::from_secs(10)).unwrap();
170 assert_eq!(colours.len(), 1, "fault {fault}");355 apply(1);
171 } else {356 reply().unwrap();
172 let (first, second) = (first.unwrap(), second.unwrap());357 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
173 assert_eq!(&colours[1..], [(first, 2), (second, 3)], "fault {fault}");358 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
359 apply(2);
360 until(&section, |event| matches!(event, Event::Failed(_)));
361 for suffix in ["-wal", "-shm"] {
362 let mut file = cache.clone().into_os_string();
363 file.push(suffix);
364 let _ = std::fs::remove_file(file);
365 }
366 if corrupt {
367 std::fs::write(&cache, [0xab; 4096]).unwrap();
368 } else {
369 std::fs::remove_file(&cache).unwrap();
370 }
371 FAILING_OPENS.store(0, Ordering::SeqCst);
372 assert!(reply().is_err());
373 assert_eq!(notebook::unwritten(), 0);
374 let pending = section.pending();
375 assert!(pending.is_err() && !would_block(pending));
376 apply(3);
377 assert!(reply().is_err());
378 let status = section.sync_status().unwrap();
379 assert_eq!((status.state(), status.queued), (SyncState::Failed, 0));
380 // The sync worker meets the lost cache and stops.
381 std::thread::sleep(Duration::from_millis(100));
382 section.events();
383 section.wake();
384 until(&section, |event| matches!(event, Event::Failed(_)));
385 assert!(section.close().is_err());
386 }
387}
388
389#[test]
390fn a_closed_cache_stays_owned_and_its_landed_commit_counts_once() {
391 let _exclusive = exclusive();
392 for close in [false, true] {
393 let directory = tempfile::tempdir().unwrap();
394 let root = directory.path();
395 let cache = seed(root);
396 let section = Section::resume(
397 root.join("Fault.one"),
398 Replica::open(&cache).unwrap(),
399 true,
400 || {},
401 )
402 .unwrap();
403 let space = section.pages().unwrap()[0].0;
404 let (replied, replies) = mpsc::channel::<Result<u64, String>>();
405 let reply = || {
406 replies
407 .recv_timeout(Duration::from_secs(10))
408 .unwrap()
409 .unwrap()
410 };
411 let apply = |value| {
412 let replied = replied.clone();
413 section
414 .apply_with("Fixture", colour(space, value), move |result| {
415 replied
416 .send(result.map_err(|error| error.to_string()))
417 .unwrap();
418 })
419 .unwrap();
420 };
421 apply(1);
422 let seeded = reply();
423 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
424 ARMED.store(SYNC_AFTER | KEEP_WAL, Ordering::SeqCst);
425 // A caller blocking on an edit whose commit fails hears at once, though it landed.
426 let asked = Instant::now();
427 assert!(
428 section
429 .replica()
430 .apply("Fixture", colour(space, 2))
431 .is_err()
432 );
433 assert!(asked.elapsed() < Duration::from_secs(5));
434 // Waits after the commit the closed cache holds, in the same batch.
435 apply(3);
436 until(&section, |event| matches!(event, Event::Failed(_)));
437 // Nobody else takes the cache to publish and acknowledge that commit meanwhile.
438 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
439 let open = if close {
440 section.close().unwrap();
441 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
442 None
443 } else {
444 Some(section)
445 };
446 FAILING_OPENS.store(0, Ordering::SeqCst);
447 let third = reply();
448 if let Some(section) = open {
449 // Reread once the cache reopens, the section is the queue.
450 let deadline = Instant::now() + Duration::from_secs(10);
451 while !section.pending().is_ok_and(|pending| pending.len() == 3) {
452 assert!(Instant::now() < deadline);
453 std::thread::sleep(Duration::from_millis(10));
454 }
455 assert_eq!(section.page(space).unwrap().color, Some(3));
456 section.close().unwrap();
457 } else {
458 let deadline = Instant::now() + Duration::from_secs(10);
459 while Replica::open(&cache).is_err() {
460 assert!(Instant::now() < deadline);
461 std::thread::sleep(Duration::from_millis(10));
174 }462 }
175 }463 }
464 let stored = colours(&cache);
465 assert_eq!(stored.len(), 3, "{stored:?}");
466 assert_eq!(
467 (stored[0], stored[1].1, stored[2]),
468 ((seeded, 1), 2, (third, 3))
469 );
176 }470 }
177}471}
472
473#[test]
474fn closed_crash_child() {
475 let (Some(root), Some(fault)) = (
476 std::env::var_os("CLOSED_CRASH_CHILD"),
477 std::env::var("COMMIT_FAULT").ok(),
478 ) else {
479 return;
480 };
481 install();
482 let root = Path::new(&root);
483 let replica = Replica::open(root.join("replica.sqlite")).unwrap();
484 let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap();
485 let space = section.pages().unwrap()[0].0;
486 let (replied, replies) = mpsc::channel();
487 let replied = move |result: Result<u64, Error>| replied.send(result.is_ok()).unwrap();
488 let apply = |value, replied: Box<dyn FnOnce(Result<u64, Error>) + Send>| {
489 section
490 .apply_with("Fixture", colour(space, value), replied)
491 .unwrap();
492 };
493 apply(1, Box::new(replied));
494 assert!(replies.recv_timeout(Duration::from_secs(10)).unwrap());
495 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
496 ARMED.store(fault.parse().unwrap(), Ordering::SeqCst);
497 apply(2, Box::new(drop));
498 until(&section, |event| matches!(event, Event::Failed(_)));
499 apply(3, Box::new(drop));
500 std::thread::sleep(Duration::from_millis(100));
501 // Ends as a crash would, the cache still closed.
502 std::process::exit(0);
503}
504
505#[test]
506fn a_crash_while_the_cache_is_closed_leaves_what_landed() {
507 for (fault, landed) in [(SYNC_BEFORE, false), (SYNC_AFTER | KEEP_WAL, true)] {
508 let directory = tempfile::tempdir().unwrap();
509 let root = directory.path();
510 let cache = seed(root);
511 let output = std::process::Command::new(std::env::current_exe().unwrap())
512 .args(["--exact", "closed_crash_child", "--nocapture"])
513 .env("CLOSED_CRASH_CHILD", root)
514 .env("COMMIT_FAULT", fault.to_string())
515 .output()
516 .unwrap();
517 assert!(output.status.success(), "{output:?}");
518 let colours: Vec<_> = colours(&cache)
519 .into_iter()
520 .map(|(_, colour)| colour)
521 .collect();
522 assert_eq!(
523 colours,
524 if landed { vec![1, 2] } else { vec![1] },
525 "fault {fault}"
526 );
527 }
528}
529
530/// A section, offline, over a seeded cache in `root`, with its first page's space.
531fn opened(root: &Path) -> (PathBuf, Section, onestore::ExGuid) {
532 let cache = seed(root);
533 let section = Section::resume(
534 root.join("Fault.one"),
535 Replica::open(&cache).unwrap(),
536 true,
537 || {},
538 )
539 .unwrap();
540 let space = section.pages().unwrap()[0].0;
541 (cache, section, space)
542}
543
544#[test]
545fn edits_wait_for_the_reread_a_refused_edit_needs_while_the_cache_is_closed() {
546 let _exclusive = exclusive();
547 for close in [false, true] {
548 let directory = tempfile::tempdir().unwrap();
549 let root = directory.path();
550 let (cache, section, space) = opened(root);
551 let (replied, replies) = mpsc::channel::<(u32, Result<u64, String>)>();
552 let apply = |space, value| {
553 let replied = replied.clone();
554 section
555 .apply_with("Fixture", colour(space, value), move |result| {
556 let result = result.map_err(|error| error.to_string());
557 replied.send((value, result)).unwrap();
558 })
559 .unwrap();
560 };
561 let reply = || replies.recv_timeout(Duration::from_secs(10)).unwrap();
562 apply(space, 1);
563 let seeded = reply().1.unwrap();
564 FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
565 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
566 // A page a caller blocks on is refused, its commit never landing.
567 let creation = PageCreation::new(None, Some("Refused"), "Fixture").unwrap();
568 let made = creation.space();
569 let create = Edit {
570 at: 133_000_000_000_000_100,
571 ops: vec![Op::Section(SectionOp::Create(creation))],
572 };
573 assert!(section.replica().apply("Fixture", create).is_err());
574 // Until the cache reopens to reread the section, edits wait to apply, one on the
575 // refused page among them.
576 apply(made, 2);
577 apply(space, 3);
578 std::thread::sleep(Duration::from_millis(1500));
579 assert!(replies.try_recv().is_err());
580 assert_eq!(section.page(space).unwrap().color, Some(1));
581 assert!(would_block(section.written()));
582 assert_eq!(notebook::unwritten(), 2);
583 let open = if close {
584 section.close().unwrap();
585 None
586 } else {
587 Some(section)
588 };
589 FAILING_OPENS.store(0, Ordering::SeqCst);
590 // A host reading the section steadily, as one typing does, keeps no retry back.
591 let deadline = Instant::now() + Duration::from_secs(10);
592 let mut answered = Vec::new();
593 while answered.len() < 2 {
594 assert!(Instant::now() < deadline);
595 if let Some(section) = &open {
596 let _ = section.page(space);
597 }
598 answered.extend(replies.recv_timeout(Duration::from_millis(300)));
599 }
600 let mut answered: [_; 2] = answered.try_into().unwrap();
601 answered.sort_by_key(|(value, _)| *value);
602 let [(_, dependent), (_, kept)] = answered;
603 assert!(dependent.is_err(), "{dependent:?}");
604 let kept = kept.unwrap();
605 if let Some(section) = open {
606 // The reread tells the host to reload what it shows.
607 until(
608 &section,
609 |event| matches!(event, Event::Changed(spaces) if spaces.contains(&made)),
610 );
611 assert!(
612 section
613 .pages()
614 .unwrap()
615 .iter()
616 .all(|(listed, ..)| *listed != made)
617 );
618 assert_eq!(section.page(space).unwrap().color, Some(3));
619 section.close().unwrap();
620 }
621 let deadline = Instant::now() + Duration::from_secs(10);
622 while Replica::open(&cache).is_err_and(|error| error.busy()) {
623 assert!(Instant::now() < deadline);
624 std::thread::sleep(Duration::from_millis(10));
625 }
626 assert_eq!(colours(&cache), [(seeded, 1), (kept, 3)]);
627 }
628}
629
630#[test]
631fn a_cache_whose_owner_lock_cannot_be_taken_stays_held_by_its_warm_connection() {
632 let _exclusive = exclusive();
633 let directory = tempfile::tempdir().unwrap();
634 let root = directory.path();
635 let (cache, section, space) = opened(root);
636 let (replied, replies) = mpsc::channel();
637 let apply = |value| {
638 let replied = replied.clone();
639 section
640 .apply_with("Fixture", colour(space, value), move |result| {
641 replied
642 .send(result.map_err(|error| error.to_string()))
643 .unwrap();
644 })
645 .unwrap();
646 };
647 let reply = || {
648 replies
649 .recv_timeout(Duration::from_secs(10))
650 .unwrap()
651 .unwrap()
652 };
653 apply(1);
654 let seeded = reply();
655 let mut lock = cache.clone().into_os_string();
656 lock.push("-lock");
657 std::fs::create_dir(&lock).unwrap();
658 ARMED.store(SYNC_BEFORE, Ordering::SeqCst);
659 apply(2);
660 until(&section, |event| matches!(event, Event::Failed(_)));
661 std::thread::sleep(Duration::from_millis(1500));
662 assert!(replies.try_recv().is_err());
663 assert!(Replica::open(&cache).is_err_and(|error| error.busy()));
664 std::fs::remove_dir(&lock).unwrap();
665 let second = reply();
666 section.close().unwrap();
667 assert_eq!(colours(&cache), [(seeded, 1), (second, 2)]);
668}
crates/notebook/tests/support/vfs.rs+24-7
...@@ -1,6 +1,6 @@...@@ -1,6 +1,6 @@
1//! A VFS over the platform's, made the process's default by `install`, that fails what a1//! A VFS over the platform's, made the process's default by `install`, that fails what a
2//! test arms: once, the WAL's sync or commit frame or the next database open; or every WAL2//! test arms: once, the WAL's sync or commit frame; the replica's next `FAILING_OPENS` opens;
3//! write of the database `FAILING` names, for as long as it names it.3//! or every WAL write of the database `FAILING` names, for as long as it names it.
4#![allow(dead_code)]4#![allow(dead_code)]
55
6use rusqlite::ffi;6use rusqlite::ffi;
...@@ -19,8 +19,6 @@ pub const SYNC_BEFORE: u8 = 1;...@@ -19,8 +19,6 @@ pub const SYNC_BEFORE: u8 = 1;
19pub const SYNC_AFTER: u8 = 2;19pub const SYNC_AFTER: u8 = 2;
20/// The commit frame's header is never written.20/// The commit frame's header is never written.
21pub const WRITE_COMMIT: u8 = 4;21pub const WRITE_COMMIT: u8 = 4;
22/// The next open of a database fails, as the cold reopen after a failed commit.
23pub const OPEN: u8 = 8;
24/// After a fault, the next sync fails, as the checkpoint closing the warm connection makes,22/// After a fault, the next sync fails, as the checkpoint closing the warm connection makes,
25/// which keeps the WAL.23/// which keeps the WAL.
26pub const KEEP_WAL: u8 = 16;24pub const KEEP_WAL: u8 = 16;
...@@ -29,12 +27,15 @@ pub static ARMED: AtomicU8 = AtomicU8::new(0);...@@ -29,12 +27,15 @@ pub static ARMED: AtomicU8 = AtomicU8::new(0);
29/// Between a fault and the next database open, as the replica reopens the cache cold.27/// Between a fault and the next database open, as the replica reopens the cache cold.
30static FIRED: AtomicBool = AtomicBool::new(false);28static FIRED: AtomicBool = AtomicBool::new(false);
31pub static WAL_WRITES_AFTER_FAULT: AtomicUsize = AtomicUsize::new(0);29pub static WAL_WRITES_AFTER_FAULT: AtomicUsize = AtomicUsize::new(0);
30/// How many database opens on a replica's own threads still fail, as the cold reopen after a
31/// failed commit can, of the database `FAILING` names where it names one; a test's opens never do.
32pub static FAILING_OPENS: AtomicUsize = AtomicUsize::new(0);
32static REAL: OnceLock<usize> = OnceLock::new();33static REAL: OnceLock<usize> = OnceLock::new();
33/// The platform's WAL file methods, the shim's copy of them, and the copy failing every write.34/// The platform's WAL file methods, the shim's copy of them, and the copy failing every write.
34static WAL: OnceLock<(usize, usize, usize)> = OnceLock::new();35static WAL: OnceLock<(usize, usize, usize)> = OnceLock::new();
35/// While set, every write to this database's WAL fails, for WALs opened while it was set.36/// While set, every write to this database's WAL fails, for WALs opened while it was set.
36pub static FAILING: Mutex<Option<PathBuf>> = Mutex::new(None);37pub static FAILING: Mutex<Option<PathBuf>> = Mutex::new(None);
37/// Held by a test for as long as it sets `FAILING`, which one test sets at a time.38/// Held by a test for as long as it arms a fault, which one test does at a time.
38pub static EXCLUSIVE: Mutex<()> = Mutex::new(());39pub static EXCLUSIVE: Mutex<()> = Mutex::new(());
3940
40fn fire(fault: u8) -> bool {41fn fire(fault: u8) -> bool {
...@@ -58,8 +59,24 @@ unsafe extern "C" fn open(...@@ -58,8 +59,24 @@ unsafe extern "C" fn open(
58) -> c_int {59) -> c_int {
59 if flags & ffi::SQLITE_OPEN_MAIN_DB != 0 {60 if flags & ffi::SQLITE_OPEN_MAIN_DB != 0 {
60 FIRED.store(false, Ordering::SeqCst);61 FIRED.store(false, Ordering::SeqCst);
61 if fire(OPEN) {62 let named = unsafe { CStr::from_ptr(name) }
62 FIRED.store(false, Ordering::SeqCst);63 .to_string_lossy()
64 .into_owned();
65 let chosen = FAILING
66 .lock()
67 .unwrap()
68 .as_ref()
69 .is_none_or(|database| named.starts_with(&*database.to_string_lossy()));
70 if chosen
71 && std::thread::current()
72 .name()
73 .is_some_and(|name| name.starts_with("onestore-"))
74 && FAILING_OPENS
75 .try_update(Ordering::SeqCst, Ordering::SeqCst, |opens| {
76 opens.checked_sub(1)
77 })
78 .is_ok()
79 {
63 unsafe { (*file).pMethods = std::ptr::null() };80 unsafe { (*file).pMethods = std::ptr::null() };
64 return ffi::SQLITE_CANTOPEN;81 return ffi::SQLITE_CANTOPEN;
65 }82 }
crates/snowbound/src/manage.rs+57-12
...@@ -436,12 +436,7 @@ impl State {...@@ -436,12 +436,7 @@ impl State {
436 state.vanishing.remove(&location);436 state.vanishing.remove(&location);
437 let waiting = (state.session.as_ref())437 let waiting = (state.session.as_ref())
438 .filter(|session| session.library.location == location)438 .filter(|session| session.library.location == location)
439 .is_some_and(|session| {439 .is_some_and(|session| unsaved(&session.section) > 0);
440 session
441 .section
442 .pending()
443 .is_ok_and(|pending| !pending.is_empty())
444 });
445 let listed = (state.notebooks.iter()).find(|open| open.location == location);440 let listed = (state.notebooks.iter()).find(|open| open.location == location);
446 if let Some(library) = listed.cloned().filter(|_| gone && !waiting) {441 if let Some(library) = listed.cloned().filter(|_| gone && !waiting) {
447 state.close_notebook(&library);442 state.close_notebook(&library);
...@@ -583,10 +578,8 @@ impl State {...@@ -583,10 +578,8 @@ impl State {
583 .filter(|library| library.in_icloud())578 .filter(|library| library.in_icloud())
584 .cloned()579 .cloned()
585 .collect();580 .collect();
586 let waiting = (self.sessions()).filter(|session| {581 let waiting = (self.sessions())
587 session.library.in_icloud()582 .filter(|session| session.library.in_icloud() && unsaved(&session.section) > 0);
588 && (session.section.pending()).is_ok_and(|pending| !pending.is_empty())
589 });
590 for session in waiting {583 for session in waiting {
591 let Some(folder) = platform::settings_dir().map(|folder| folder.join("Recovery"))584 let Some(folder) = platform::settings_dir().map(|folder| folder.join("Recovery"))
592 else {585 else {
...@@ -653,8 +646,7 @@ impl State {...@@ -653,8 +646,7 @@ impl State {
653 self.persist().unwrap_or_else(|error| eprintln!("{error}"));646 self.persist().unwrap_or_else(|error| eprintln!("{error}"));
654 let open = (self.session.as_ref())647 let open = (self.session.as_ref())
655 .filter(|session| Arc::ptr_eq(&session.library, &library))648 .filter(|session| Arc::ptr_eq(&session.library, &library))
656 .and_then(|session| session.section.pending().ok())649 .map_or(0, |session| unsaved(&session.section));
657 .map_or(0, |pending| pending.len() as u64);
658 let waiting = (library.background.iter())650 let waiting = (library.background.iter())
659 .flat_map(|background| background.status())651 .flat_map(|background| background.status())
660 .map(|(_, sync)| sync.queued)652 .map(|(_, sync)| sync.queued)
...@@ -1211,6 +1203,12 @@ fn art_ops(page: &Page, art: Vec<Image>) -> Vec<PageOp> {...@@ -1211,6 +1203,12 @@ fn art_ops(page: &Page, art: Vec<Image>) -> Vec<PageOp> {
1211 ops1203 ops
1212}1204}
12131205
1206/// The changes `section` holds that its file does not yet, as many as can be told, and at least
1207/// one where none can.
1208fn unsaved(section: &notebook::session::Section) -> u64 {
1209 section.sync_status().map_or(1, |sync| sync.queued)
1210}
1211
1214#[cfg(test)]1212#[cfg(test)]
1215mod tests {1213mod tests {
1216 use super::*;1214 use super::*;
...@@ -1225,6 +1223,53 @@ mod tests {...@@ -1225,6 +1223,53 @@ mod tests {
1225 creation.dated(&date, &time).unwrap()1223 creation.dated(&date, &time).unwrap()
1226 }1224 }
12271225
1226 #[test]
1227 fn unsaved_counts_what_a_closed_cache_holds() {
1228 use std::sync::atomic::Ordering;
1229 crate::vfs::install();
1230 let _exclusive =
1231 (crate::vfs::EXCLUSIVE.lock()).unwrap_or_else(std::sync::PoisonError::into_inner);
1232 let temporary =
1233 std::env::temp_dir().join(format!("snowbound-unsaved-{}", std::process::id()));
1234 let _ = notebook::fs::remove_dir_all(&temporary);
1235 notebook::fs::create_dir_all(&temporary).unwrap();
1236 let file = temporary.join("Unsaved.one");
1237 let cache = temporary.join("unsaved.sqlite");
1238 let source = onestore::create_section("Unsaved.one", "Original", AUTHOR).unwrap();
1239 notebook::fs::write(&file, &source).unwrap();
1240 let open = || {
1241 let replica = notebook::Replica::open(&cache).unwrap();
1242 notebook::session::Section::resume(&file, replica, true, || {}).unwrap()
1243 };
1244 drop(notebook::Replica::create(&cache, &source).unwrap());
1245 let section = open();
1246 let space = section.pages().unwrap()[0].0;
1247 let colour = |colour| Edit {
1248 at: crate::filetime(),
1249 ops: vec![Op::Page {
1250 space,
1251 op: PageOp::Color(Some(colour)),
1252 }],
1253 };
1254 section.replica().apply(AUTHOR, colour(1)).unwrap();
1255 section.close().unwrap();
1256 // The cache's next commit fails, and it cannot reopen; the edit is refused.
1257 *crate::vfs::FAILING.lock().unwrap() = Some(cache.canonicalize().unwrap());
1258 crate::vfs::FAILING_OPENS.store(usize::MAX, Ordering::SeqCst);
1259 let section = open();
1260 assert_eq!(unsaved(&section), 1);
1261 assert!(section.replica().apply(AUTHOR, colour(2)).is_err());
1262 assert!(section.pending().is_err());
1263 assert_eq!(unsaved(&section), 1);
1264 *crate::vfs::FAILING.lock().unwrap() = None;
1265 crate::vfs::FAILING_OPENS.store(0, Ordering::SeqCst);
1266 section.close().unwrap();
1267 while notebook::Replica::open(&cache).is_err() {
1268 std::thread::sleep(std::time::Duration::from_millis(10));
1269 }
1270 let _ = notebook::fs::remove_dir_all(&temporary);
1271 }
1272
1228 #[test]1273 #[test]
1229 fn repeated_section_renames_follow_identity_and_keep_the_latest_label() {1274 fn repeated_section_renames_follow_identity_and_keep_the_latest_label() {
1230 let temporary =1275 let temporary =