Howard Hinnant via llvm-dev
2018-Aug-10 18:43 UTC
[llvm-dev] [cfe-dev] Filesystem has Landed in Libc++
On Aug 10, 2018, at 1:28 PM, Marshall Clow via cfe-dev <cfe-dev at lists.llvm.org> wrote:> > * The clock stuff being added in C++20 has already been discussed here.I’ve missed the discussions on file_time_type, however I thought I should throw in my opinion here before it is too late to do anything about it. I believe it is a mistake to model file_time_type with 128 bits. It would be acceptable if this was absolutely necessary to get the job done, but it isn’t. The 16 byte integer is unnecessarily expensive to get the job done. file_time_type does not need to model the full range and precision of timespec (which on 64 bit platforms is a 128 bit type). All file_time_type needs to model is the full range and precision of what the underlying file system libraries are capable of producing. The latest Linux file system is ext4 (https://en.wikipedia.org/wiki/Ext4) and is capable of nanosecond resolution. However its timestamp is only 64 bits. It has a range of approximately [1901-12-14, 2446-05-10]. Modeling ext4 would be a good design decision for libc++. libc++ could also model other file systems (Windows, macOS). All of these are based on 64 bit timestamps. Here is a file_clock, quickly thrown together, lightly tested, that models ext4: #include "date/tz.h" #include <ostream> #include <istream> namespace filesystem { struct file_clock { using duration = std::chrono::nanoseconds; using rep = duration::rep; using period = duration::period; using time_point = std::chrono::time_point<file_clock>; static constexpr bool is_steady = false; static time_point now(); template<typename Duration> static std::chrono::time_point<std::chrono::system_clock, Duration> to_sys(const std::chrono::time_point<file_clock, Duration>& t) noexcept; template<typename Duration> static std::chrono::time_point<file_clock, Duration> from_sys(const std::chrono::time_point<std::chrono::system_clock, Duration>& t) noexcept; template<typename Duration> static std::chrono::time_point<date::local_t, Duration> to_local(const std::chrono::time_point<file_clock, Duration>& t) noexcept; template<typename Duration> static std::chrono::time_point<file_clock, Duration> from_local(const std::chrono::time_point<date::local_t, Duration>& t) noexcept; // private helpers static timespec to_timespec(const time_point& t) noexcept; static time_point from_timespec(const timespec& t) noexcept; }; template <class Duration> using file_time = std::chrono::time_point<file_clock, Duration>; using file_time_type = file_clock::time_point; template <class Duration> inline std::chrono::time_point<std::chrono::system_clock, Duration> file_clock::to_sys(const std::chrono::time_point<file_clock, Duration>& t) noexcept { using namespace date; return sys_time<Duration>{t.time_since_epoch()} + (sys_days{2174_y/1/1} - sys_days{1970_y/1/1}); } template <class Duration> inline std::chrono::time_point<file_clock, Duration> file_clock::from_sys(const std::chrono::time_point<std::chrono::system_clock, Duration>& t) noexcept { using namespace date; return file_time<Duration>{t.time_since_epoch()} - (sys_days{2174_y/1/1} - sys_days{1970_y/1/1}); } template <class Duration> inline std::chrono::time_point<date::local_t, Duration> file_clock::to_local(const std::chrono::time_point<file_clock, Duration>& t) noexcept { using namespace date; return local_time<Duration>{to_sys(t).time_since_epoch()}; } template <class Duration> inline std::chrono::time_point<file_clock, Duration> file_clock::from_local(const std::chrono::time_point<date::local_t, Duration>& t) noexcept { using namespace date; return file_time<Duration>{from_sys(sys_time<Duration>{t.time_since_epoch()})}; } file_clock::time_point file_clock::now() { return from_sys(std::chrono::system_clock::now()); } template <class CharT, class Traits, class Duration> std::basic_ostream<CharT, Traits>& to_stream(std::basic_ostream<CharT, Traits>& os, const CharT* fmt, const file_time<Duration>& t) { using namespace std::chrono; const std::string abbrev("UTC"); constexpr std::chrono::seconds offset{0}; using D128 = duration<__int128, typename Duration::period>; return date::to_stream(os, fmt, file_clock::to_local(time_point_cast<D128>(t)), &abbrev, &offset); } template <class Duration, class CharT, class Traits, class Alloc = std::allocator<CharT>> std::basic_istream<CharT, Traits>& from_stream(std::basic_istream<CharT, Traits>& is, const CharT* fmt, file_time<Duration>& tp, std::basic_string<CharT, Traits, Alloc>* abbrev = nullptr, std::chrono::minutes* offset = nullptr) { using namespace date; using namespace std::chrono; using D128 = duration<__int128, typename Duration::period>; local_time<D128> lp; from_stream(is, fmt, lp, abbrev, offset); if (!is.fail()) tp = file_clock::from_local(lp); return is; } template <class CharT, class Traits, class Duration> std::basic_ostream<CharT, Traits>& operator<<(std::basic_ostream<CharT, Traits>& os, const file_time<Duration>& t) { const CharT fmt[] = {'%', 'F', ' ', '%', 'T', CharT{}}; return to_stream(os, fmt, t); } inline timespec file_clock::to_timespec(const time_point& t) noexcept { using namespace date; using namespace std::chrono; auto tp = to_sys(time_point_cast<std::chrono::duration<__int128, std::nano>>(t)); auto s = floor<seconds>(tp); timespec ts; ts.tv_sec = static_cast<decltype(ts.tv_sec)>(s.time_since_epoch().count()); ts.tv_nsec = static_cast<decltype(ts.tv_nsec)>((tp - s).count()); return ts; } inline file_clock::time_point file_clock::from_timespec(const timespec& t) noexcept { using namespace date; using namespace std::chrono; auto d = std::chrono::duration<__int128>{t.tv_sec} + nanoseconds{t.tv_nsec}; return time_point_cast<duration>(from_sys(sys_time<decltype(d)>{d})); } } // namespace filesystem #include <iostream> #include <sstream> int main() { using namespace std; using namespace date; std::cout << filesystem::file_clock::time_point::min() << '\n'; std::cout << filesystem::file_clock::now() << '\n'; std::cout << filesystem::file_clock::time_point::max() << '\n'; std::istringstream in{"2466-04-11 23:47:16.854775807"}; filesystem::file_clock::time_point tp; in >> date::parse("%F %T", tp); cout << tp << '\n'; in.clear(); in.str("1881-09-22 00:12:43.145224192"); in >> date::parse("%F %T", tp); cout << tp << '\n'; timespec ts = {15661036036, 854775807}; // or {-2785708037, 145224192} tp = filesystem::file_clock::from_timespec(ts); cout << tp << '\n'; ts = filesystem::file_clock::to_timespec(tp); cout << "{" << ts.tv_sec << ", " << ts.tv_nsec << "}\n"; using s32 = chrono::duration<int>; using ns64 = chrono::duration<long, nano>; using uns64 = chrono::duration<unsigned long, nano>; using ns128 = chrono::duration<__int128, nano>; cout << date::sys_time<s32>::min() + ns128{uns64::max()} << '\n'; cout << date::sys_time<s32>::min() + ns128{ns64::max()} + 1ns << '\n'; } It is a 64bit timestamp with nanosecond resolution that is capable of representing a superset of ext4 (about +/- 20 years on either side of the limits of ext4). It _does_ internally use __int128 for a few intermediate computations such as converting to/from a timespec. This allows it avoid overflow out near min()/max(). However, the most common operations users will encounter are simply the arithmetic involving time_point and duration, and these are strictly 64 bit (and already provided by chrono). This is what I advise for the libc++ filesystem library. I have also sent this code to the gcc developers. Consider it public domain. Howard -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180810/e87f8564/attachment.sig>
Eric Fiselier via llvm-dev
2018-Aug-11 01:35 UTC
[llvm-dev] [cfe-dev] Filesystem has Landed in Libc++
On Fri, Aug 10, 2018 at 2:43 PM Howard Hinnant via cfe-dev < cfe-dev at lists.llvm.org> wrote:> On Aug 10, 2018, at 1:28 PM, Marshall Clow via cfe-dev < > cfe-dev at lists.llvm.org> wrote: > > > > * The clock stuff being added in C++20 has already been discussed here. > > I’ve missed the discussions on file_time_type, however I thought I should > throw in my opinion here before it is too late to do anything about it. > > I believe it is a mistake to model file_time_type with 128 bits. It would > be acceptable if this was absolutely necessary to get the job done, but it > isn’t. The 16 byte integer is unnecessarily expensive to get the job done. > > file_time_type does not need to model the full range and precision of > timespec (which on 64 bit platforms is a 128 bit type). All file_time_type > needs to model is the full range and precision of what the underlying file > system libraries are capable of producing. > > The latest Linux file system is ext4 (https://en.wikipedia.org/wiki/Ext4) > and is capable of nanosecond resolution. However its timestamp is only 64 > bits. It has a range of approximately [1901-12-14, 2446-05-10]. Modeling > ext4 would be a good design decision for libc++. libc++ could also model > other file systems (Windows, macOS). All of these are based on 64 bit > timestamps. >It does seem that I overlooked the fact that EXT4 only provides 64 bits of precision despite timespec providing more. I'll look into other filesystems to see if any offer more than that. If not, perhaps I was mistaken trying to match the precision and range of timespec. Part of me is still concerned with the future, and the filesystems which are yet to exist.> > Here is a file_clock, quickly thrown together, lightly tested, that models > ext4: > > #include "date/tz.h" > #include <ostream> > #include <istream> > > namespace filesystem > { > > struct file_clock > { > using duration = std::chrono::nanoseconds; > using rep = duration::rep; > using period = duration::period; > using time_point > std::chrono::time_point<file_clock>; > static constexpr bool is_steady = false; > > static time_point now(); > > template<typename Duration> > static > std::chrono::time_point<std::chrono::system_clock, Duration> > to_sys(const std::chrono::time_point<file_clock, Duration>& t) > noexcept; > > template<typename Duration> > static > std::chrono::time_point<file_clock, Duration> > from_sys(const std::chrono::time_point<std::chrono::system_clock, > Duration>& t) noexcept; > > template<typename Duration> > static > std::chrono::time_point<date::local_t, Duration> > to_local(const std::chrono::time_point<file_clock, Duration>& t) > noexcept; > > template<typename Duration> > static > std::chrono::time_point<file_clock, Duration> > from_local(const std::chrono::time_point<date::local_t, Duration>& > t) noexcept; > > // private helpers > > static > timespec > to_timespec(const time_point& t) noexcept; > > static > time_point > from_timespec(const timespec& t) noexcept; > }; > > template <class Duration> > using file_time = std::chrono::time_point<file_clock, Duration>; > > using file_time_type = file_clock::time_point; > > template <class Duration> > inline > std::chrono::time_point<std::chrono::system_clock, Duration> > file_clock::to_sys(const std::chrono::time_point<file_clock, > Duration>& t) noexcept > { > using namespace date; > return sys_time<Duration>{t.time_since_epoch()} + > (sys_days{2174_y/1/1} - > sys_days{1970_y/1/1}); > } > > template <class Duration> > inline > std::chrono::time_point<file_clock, Duration> > file_clock::from_sys(const > std::chrono::time_point<std::chrono::system_clock, Duration>& t) noexcept > { > using namespace date; > return file_time<Duration>{t.time_since_epoch()} - > (sys_days{2174_y/1/1} - > sys_days{1970_y/1/1}); > } > > template <class Duration> > inline > std::chrono::time_point<date::local_t, Duration> > file_clock::to_local(const std::chrono::time_point<file_clock, > Duration>& t) noexcept > { > using namespace date; > return local_time<Duration>{to_sys(t).time_since_epoch()}; > } > > template <class Duration> > inline > std::chrono::time_point<file_clock, Duration> > file_clock::from_local(const std::chrono::time_point<date::local_t, > Duration>& t) noexcept > { > using namespace date; > return > file_time<Duration>{from_sys(sys_time<Duration>{t.time_since_epoch()})}; > } > > file_clock::time_point > file_clock::now() > { > return from_sys(std::chrono::system_clock::now()); > } > > template <class CharT, class Traits, class Duration> > std::basic_ostream<CharT, Traits>& > to_stream(std::basic_ostream<CharT, Traits>& os, const CharT* fmt, > const file_time<Duration>& t) > { > using namespace std::chrono; > const std::string abbrev("UTC"); > constexpr std::chrono::seconds offset{0}; > using D128 = duration<__int128, typename Duration::period>; > return date::to_stream(os, fmt, > file_clock::to_local(time_point_cast<D128>(t)), > &abbrev, &offset); > } > > template <class Duration, class CharT, class Traits, class Alloc > std::allocator<CharT>> > std::basic_istream<CharT, Traits>& > from_stream(std::basic_istream<CharT, Traits>& is, const CharT* fmt, > file_time<Duration>& tp, > std::basic_string<CharT, Traits, Alloc>* abbrev = nullptr, > std::chrono::minutes* offset = nullptr) > { > using namespace date; > using namespace std::chrono; > using D128 = duration<__int128, typename Duration::period>; > local_time<D128> lp; > from_stream(is, fmt, lp, abbrev, offset); > if (!is.fail()) > tp = file_clock::from_local(lp); > return is; > } > > template <class CharT, class Traits, class Duration> > std::basic_ostream<CharT, Traits>& > operator<<(std::basic_ostream<CharT, Traits>& os, const > file_time<Duration>& t) > { > const CharT fmt[] = {'%', 'F', ' ', '%', 'T', CharT{}}; > return to_stream(os, fmt, t); > } > > inline > timespec > file_clock::to_timespec(const time_point& t) noexcept > { > using namespace date; > using namespace std::chrono; > auto tp = to_sys(time_point_cast<std::chrono::duration<__int128, > std::nano>>(t)); > auto s = floor<seconds>(tp); > timespec ts; > ts.tv_sec > static_cast<decltype(ts.tv_sec)>(s.time_since_epoch().count()); > ts.tv_nsec = static_cast<decltype(ts.tv_nsec)>((tp - s).count()); > return ts; > } > > inline > file_clock::time_point > file_clock::from_timespec(const timespec& t) noexcept > { > using namespace date; > using namespace std::chrono; > auto d = std::chrono::duration<__int128>{t.tv_sec} + > nanoseconds{t.tv_nsec}; > return > time_point_cast<duration>(from_sys(sys_time<decltype(d)>{d})); > } > > } // namespace filesystem > > #include <iostream> > #include <sstream> > > int > main() > { > using namespace std; > using namespace date; > std::cout << filesystem::file_clock::time_point::min() << '\n'; > std::cout << filesystem::file_clock::now() << '\n'; > std::cout << filesystem::file_clock::time_point::max() << '\n'; > std::istringstream in{"2466-04-11 23:47:16.854775807"}; > filesystem::file_clock::time_point tp; > in >> date::parse("%F %T", tp); > cout << tp << '\n'; > in.clear(); > in.str("1881-09-22 00:12:43.145224192"); > in >> date::parse("%F %T", tp); > cout << tp << '\n'; > timespec ts = {15661036036, 854775807}; // or {-2785708037, > 145224192} > tp = filesystem::file_clock::from_timespec(ts); > cout << tp << '\n'; > ts = filesystem::file_clock::to_timespec(tp); > cout << "{" << ts.tv_sec << ", " << ts.tv_nsec << "}\n"; > using s32 = chrono::duration<int>; > using ns64 = chrono::duration<long, nano>; > using uns64 = chrono::duration<unsigned long, nano>; > using ns128 = chrono::duration<__int128, nano>; > cout << date::sys_time<s32>::min() + ns128{uns64::max()} << '\n'; > cout << date::sys_time<s32>::min() + ns128{ns64::max()} + 1ns << > '\n'; > } > > It is a 64bit timestamp with nanosecond resolution that is capable of > representing a superset of ext4 (about +/- 20 years on either side of the > limits of ext4). It _does_ internally use __int128 for a few intermediate > computations such as converting to/from a timespec. This allows it avoid > overflow out near min()/max(). However, the most common operations users > will encounter are simply the arithmetic involving time_point and duration, > and these are strictly 64 bit (and already provided by chrono). > > This is what I advise for the libc++ filesystem library. I have also sent > this code to the gcc developers. Consider it public domain. > > Howard > > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180810/7d46c703/attachment.html>
Howard Hinnant via llvm-dev
2018-Aug-11 02:41 UTC
[llvm-dev] [cfe-dev] Filesystem has Landed in Libc++
On Aug 10, 2018, at 9:35 PM, Eric Fiselier <eric at efcs.ca> wrote:> > Part of me is still concerned with the future, and the filesystems which are yet to exist. >Me too. But it is best to target modern systems when targeting future systems adds an unnecessary cost. When future systems come into being, it is likely because future hardware is making those future systems practical. E.g. nanosecond precision file systems were not produced prior to the widespread adoption of 64 bit hardware. Mainly because they were just too expensive on 32 bit hardware. In the future, we will have a better shot at dealing with that future. The std::lib we write today will have to evolve, no matter what we do today. Future proof where it is practical to do so, and don’t where it isn’t. Howard -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180810/5e73a64f/attachment.sig>