Was not passing stopped state due to undefined play but need to do this to let last play actually be scrobbled.
Refactor jellyfin session validation and types for MemorySource to correctly allow processing player state with no plays
* Default disallow any activity not from a library of collectionType 'music'
* Allow user config for allow/block by library name
* Allow user config for allowing additional library collection types
* Update docs and example config
SDK types say AlbumArtists etc. should be defined but #200 shows that's not always the case, for some reason.
Check for undefined value on playback data that requires additional transformation to prevent errors.
Fixes#200
Fixes a lot of headaches:
* vega is more up to date and extensible
* no more hacky regex for reference names
* complete control over parsing and formatting (const as enum)
* does not duplicate typescript as dependency
* should properly throw errors on schema generation instead of just logging to console
* move schema static files into programmatic creation instead of package scripts
* fix types being removed in prod which caused schema generation errors
Not enough to include it in both dependencies and devDependencies -- it needs to be NOT in devDependencies to prevent `npm ci --omit=dev` from purging it.
* Add suite for testing client/source parsing and validation
* Additionally, use example configs for two-birds-one-stone checking they are valid
* Refactor sources init as separate step from component building
* Use runtime-generated schemas
* Ensures validated schemas always match code
* Can use schemas for individual source/clients without having to generate files
* catch invididual validation errors and log instead of crashing
* refactor validation error output into chained error instead of as individual log statements to improve clarity
* Swtch to devcontainer for node 18.x and remove terminal/launch settings for version (removes nvm dependency in container)
* Update eslint config to use proper config for include/ignore files
* Add spec for mocha and mocha test runner extension + settings
* Fix documentation and jellyfin example to match code
* Add tests for jellyfin api config parsing and valid activities
* Fix jellyin api ENV naming to match code
* Fix config parsing missing lowercase transform
Fixes#196
* All required files in the same file by using .actrc
* Include example .secrets and .env
* Update README with usage
* Refactor actTest to make docker build steps optional with ENV
* Add refreshMinInterval to prevent hammering upstream services
* Refactor refresh logic to be simpler and account for backlogged tracks
* (test): Improve play generation utils for testing
* (test): Refactor testing for upstream scrobble refreshing to use more actual scrobbler class behavior
* Move file parsing to own function
* Add some typings to file parsing to help with development (typings are bad)
* Defensively check structure of formdata for nested arrays since Plex seems to waffle on how it sends data (fixes#191)
* Add logging for all files found to help with future end-user debugging
express-static prefers directories over files when using the extensions
option, meaning an extensionless route will always route to the
directory instead of the file (expressjs/serve-static#138).
* Ensure pre-release tags don't also get latest tag
* Disable alpine builds
* Only increases testing and build complexity and there has been no feedback from users about needing it
* Aligns with "normal" default branch usage better, allows dependabot to actually run
* Change image publishing to push tagged images as 'latest' and master images as `edge`
* Implement new jellyfin source using JF typescript api client library for communication
* Use either API Key or username/password
* Implement real-time scrobble monitoring
* Implement feature parity filters from old jellyfin source (users) and add devices filters
* Refactor documentation for new configuration with api and add migration steps for webhook users
* Add deprecation warning to jellyfin webhook source
* Move recentScrobbles and generic refresh logic into abstract class
* New abstract function for scrobbler children to implement to get scrobbler-specific play lists on refresh
* Implement shouldRefereshScrobble with logging for better insight into why or why not a scrobbler refreshed upstream scrobbles
Since docs are now generated alongside builds and hosted on the app server there is no longer a barrier to using docusaurus docs as the main documentation presentation. Any user looking for docs can use the GH Pages hosted site or a "versioned" docs site hosted on their MS app. Due to this we can fully commit to docusaurus and take advantage of MDX layouts to simplify docs.
* Replace env/file configs headings in configuration with MDX tabs to make reading easier
* Remove superfluous TOCs
* Replace internal links in repo readme with links to docsite
* setup.sh takes care of building generated sources for project/docsite and building flatpak app
* Modify flatpak manifest to include two generated sources files and build docsite files
* Update flatpak docs with script usage and generated sources changes, move to docusauraus doc site
* Add npm scripts for installing/building docusaurus from main project
* Set base url based on ENV
* Build docsite in docker image
* Add link to doc in dashboard and link to github
* Update nodejs install instructions to include doc build command
* Update GH pages workflow to use correct base url
* Add hint page to main app when docs are not built
* Use lower headers so docusaurs sidebar TOC is more accurate
* Specify header levels for config/FAQ to make sidebar more concise
* Wrap inline TOC in details block -- so docusaurs users don't have to see it twice but its still available to github viewers
Sporadic DNS issues with alpine have caused a number of issues #160#126#134
Use debian as default since it has been proven to be stable. The image size increase is worth squashing these issues.
* Use superdiff to diff PlayObject lists and detect changes as well as append/prepend scenarios
* Replace YTM recently played logic with list diffing, only accept prepend-validated lists
* On non-prepend scenarios replace existing recently played and log human readable diff
* Patch youtube-music-ts-api to use updated cookies from response and provide a callback on update
* Implement currentCreds/build init data and read from MS-updated creds if available
* Write to currentCreds when ytm-ts-api invokes auth update callback and optionally log what parts changed based on config options
* Update Vite dep
* Use inlineViteConfig for vite-express middleware and detect correct base string
* Refactor vite.config.ts to detect correct base string
* Move code into testable function
* Fix port erasure when url uses standard port but it is explicitly specified
* Simplify building default url and fix using undefined mainPort variable
* Fix port erasure when https protocol is explicitly specified
* Refactor localUrl usage as URL instead of string
* Implement pathname joining on URL to make handling trailing/leading slashes take place in nodejs lib
* Implements tests for URL building/manipulation
* Increase default backlog limits to maximum pagination supported by each source
* Add scrobbleBacklogCount config property so user can explicitly set # of tracks to backlog
* Move some debug => verbose for non-repeating logs that are relevant to state, notifications, and scrobbling
* Move some verbose => debug that were repeating and making verbose level noisy
If a user does not need the dashboard or has security concerns about ingress and cannot control their hosting environment this enables reducing attack surface.
Closes#150
Error Cause (https://github.com/tc39/proposal-error-cause) was finalized and introduced in ES2022 so we don't need to ponyfill this anymore. Still using the helpful helper functions thought.
* Refactor SOT to be a type (may have more in the future)
* Set player stale/orphan interval based on SOT type
* Add note in log when initializing MemorySource which does not use player as SOT
* Add tooltip to UI player when it is not SOT
Migrating to @spotify/spotify-web-api-ts-sdk requires that the credentials we provide have valid expires_in/expires data and grant type. Make this migration easier by writing that data now so users upgrading later most likely do not need to re-authenticate.
* Actually remove device if it fails after X retries
* Properly close controllers/platform/client to prevent uncaught errors from castv2
* Move manually configured devices to device discovery function so they can be reconnected on heartbeat (if previously removed)
* Implement interfaces for response shape and getters for data
* Parse version data from response and use ping endpoint to return info on connection test
* Fix throwing auth/connection errors to include actual cause along with more human-friendly parsed error info
* Build error causes based on known error response types and expected maloja error body
* Only retry requests if error is network-related
* Set upstream error as NOT showstopping if we find expected error structure and status is less than 500
* Better maloja error parsing and logging
* Switch to lower-level mdns library for more control during debugging
* Use avahi-browse with host avahi-daemon to circumvent musl (alpline) issues with mdns resolution https://johnsiu.com/blog/alpine-linux/
* Add transportId shortname to logger label for better differentiation of "per session" apps
* Fix handling of MS player deletion when pruning apps by checking for other valid apps before deletion
* Refactor credit parsing into two stages: joiner separation and credits parsing
* Break credit parsing into wrapped vs. non-wrapped for simpler regexes
* Implement suffix matching after credits
* Add tests for wrapped vs. non-wrapped credits and with suffixes
* Break up initialize into build data => check connection => check auth
* Implement stub methods in AbstractSource with default return for steps
* Refactor existing initialize overrides into steps for each source
* Move (re)start to same location on status card as client and change verbage to be the same
* Fix api behavior to use correct poll function so that backlog is triggered
* Consolidate playerDelete event emitter to deletePlayer function
* Consolidate logging for player deletion to deletePlayer function
* Add removal reason when transferring player state for single-platform sources
The assumption is that single platform sources (like spotify) can only play from one platform at a time. If they are smart (like Spotify) the "new" platform picks up where the old platform left off WRT playing music. This commit implements the same behavior for MS player state by transferring existing state from the existing (previous) platform to the newly discovered platform.
* Track datetime player "completes" play at
* Add scrobble datetime SOC metadata
* Log SOC used when printing play datetime
* Use hinted SOC scrobble datetime when comparing existing scrobbles
* Use hinted SOC scrobble datetime when building scrobbler client scrobble payload
* Hint Spotify backlog plays use END play date for scrobble SOC
* Refactor using 'close' boolean to 'match' granularity
* Makes using granularity for future logic easier
* Easier logging for granularity in summary
* Remove intermediate temporal functions in classes for DRY and so we can use comparison results
* Add Time Detail to match breakdown for more visibility during logging
* Fix undefined token inclusion caused by token lists with differing lengths
* Always compare longer to shorter string so sameness parameter order is invariant
* Add comments to make logic easier to understand
* Add tests to test new functionality
* Move logging for init/auth into client instead of builder class
* Move startup from builder class to main backend and use initScrobbleMonitoring() instead of calling init/auth individually
Paves the way for manual restarting #114
If player is calculated as playing and tracks have positional data then start and end positions can be fudged to completion if jitter is small enough
* At beginning of new track if position is within 3 seconds of start of track fudge to 1 second
* At end of old track if position is within 3 seconds of end track fudge to completion
This can be achieved with high confidence due to smart polling near end of tracks added in 1d84f07a40
If player has an absolute position and play has duration we can determine if polling interval will occur after play would end. In this scenario we can shorten interval to before play ends to get more accurate data about play state close to end of track and report more accurate play time.
* Wake up timestamp was based on ts AFTER polling finished + interval. This means real-time interval between polling drifted by the amount of time it takes to execute polling loop. Fixed by getting ts at beginning of loop instead of at end.
* Implement interval function that can be overriding by child classes
* Format interval to make floats less ugly
* Implement abstract playToClientPayload function to make future testing easier
* Refactor play object to include album artist in data
* Refactor scrobble client payloads to include album artist, if applicable
* Discriminate between proper errors (permissions) and network errors
* Fix return value and playbackState flag set during post-auth hook
* Improve error logging using cause and upstream error
* Fix undefined destructure when device is not present in playingNow payload
* Implement type/guard for node network error
* Refactor auth in sources & clients to flag if error is network or api based
* Retry auth on heartbeat if auth failure was caused by network
* Implement initial tests for testing client auth
* Fix tests hanging due to running async functions
* Store logs by levels, fixes issue where rarer warn/info are pushed out of buffer by frequent debug/verbose
* Use FixedSizeList instead of homegrown buffer
* Implement API and ui controls for setting log limit
* Implement backend api endpoints for dead letter CRUD
* Include dead letter queue count in client status data
* Implement dead letter client view
* Partially working retry/remove actions for client (request is OK, RTK not working for update yet)
* Fix repeat end position percent test (should be 85% not 15%)
* Fix current play/range not being reset correctly on repeat
* Add reportedTS param to facilitate testing
* Use running/idle for scrobble client to indicate processing status
* Change polling sources status to polling/idle to better reflect behavior
* Add status indicator color to statuses
* Delay scrobble based on last scrobble attempt TS
* rename abstract scrobble func to doScrobble
* Implement scrobble() in abstract client and track last scrobble attempt time, delay based on arg or client property
* This way first scrobble isn't delayed and we only delay if subsequent scrobble attempts are close in time
* Make scrobble processing sleep time on empty queue configurable
* Implemented logic for processing and removing scrobbles from dead letter queue
* Only retry scrobbles under X retries based on config or argument
* Add removal-only function (for later use with UI)
* Move scrobble checking into client (out of client manager) for single responsibility
* Implement scrobble queue/deadletter
* Less chance of duplicate processing since sources can't scrobble concurrently
* Better throttling since multiple sources can't scrobble concurrently
* When scrobbling fails we can retry on delay just like polling and *keep* the failed scrobbles for later retry rather than dropping them
* Deadletter (to be fully implemented) for scrobbles that fail multiple times or are signalled as not upstream api issues by client
Wait until player gets a new play or becomes stale before trying to discover the (previous) play so that we store a more accurate amount of time the user actually listened to that play
When looking for dups and finding a fuzzy timestamp match, close title, and mismatch (but some matched) multiple artist -- we give a bonus to artist score since its likely a dup.
* Add text/* wildcard so any text-prefixed content-type is valid (Jellyfin should always be sending json anyway)
* Add logging WARN when body is empty
Fixes#101
* QoL improvement -- Instead of needing to rename a config file so it is not read by MS a user can now use the 'enable' property in all source/client configs to enable or disable loading the config
* Added property to all config examples
* Use lastfm 'nowPlaying' property in user listens for MS player implementation
* Do not use MS player as source of truth -- continue to use lfm listen history
Will not be used by listenbrainz source for actual scrobbling so make logs less noisy by suppressing what would normally be logging on newely added tracks from player
* Allows users to define a base URL to be used for default redirect URI generation
* Convenience mostly for docker users so they do not have to explicitly define redirect uri for each source/client
* Use in UI start up logging hints
files in node_module can have wierd ownership based on how a package owner published the package. If this occurs a rootless container build system could potentially have issues with subui/gid mapping for non-standard (high number) ids.
During image build recursively chown all node_modules files to normalize ownership and avoid this problem.
Fixes#96
* Due to increased scrobbled thresholds in 590610022d we need to pol more frequently to make sure more of the track is accurately monitored
* Decreased default from 30 to 10 seconds
* Decreased max interval from 60 to 30 seconds
* Better use of header
* Show name on separate line
* Only show client/source if ambiguous
* Move status into right side of header
* Reduce padding/margin on card grid
* Reduce padding on card headers and bodies
* Improve log experience
* Reduce log font size and convert to mono
* Use proper log message parsing to remove dangerous html set
* Check for stale/orphaned/dead in independent loop so they are not dependent on getting source updates to update
* Only emit events if player state has updated
* Create and implement api interface for player state data
* Serialize memory source players to object indexed by platform id
* Refactor event emitters to separate event data better from event metadata
* Update source slice when player update event is broadcast
Addresses #5 and #65 -- in default config use-cases where:
* request/polling retry attempts are low
* source was initially OK (auth OK, initial communication OK)
* and there is a non-auth related *upstream* communication issue
there is no fallback path for restarting source polling after upstream has normalized.
This feat implements a scheduled heartbeat task that attempts to restart polling every 20 minutes if this scenario is valid so that a good source config will recover without user intervention.
* Update flatpak build instructions to use build
* Add run command to readme
* Update default config to keep web port backwards compatible
* Fix flatpak application name in run command in installation docs
* Rename to backend to better reflect what it is
* Move server functionality out of entry file and into own server subdirectory
* Rename 'apis' folder to 'vendor' to reflect usage and move into common
* Rename 'clients' folder to 'scrobblers' reflect usage and prevent ambiguity with client CRA folder
* Changed logo color to be easy to view in both dark/light modes (better color in general, imo)
* Changed and cleaned up head/meta/manifest info and logos for docs and frontend
* If user tried to start polling while polling was already occuring it caused an infinite while loop
* Add more signals for stopping polling for a restart use case
* Result and criteria are returned from methods
* Logging method to summarize results
* Log results for jellyfin if using debug mode or explicit config option
* Will eventually migrate all non-source specific properties to this new property to simplify data config
* Add logPlayerState to allow configurable player state output to DEBUG logging
* Implements generic Player state tracking when using MemorySource
* State tracking will replace processRecentPlays and all its data structures
* Tracks reported player position OR real-time listened when position not available
* Records listened sessions for more exhaustive "listen to" duration and eventually comparing incoming data playedAt for dups
* Uses own tracked data to calculate most likely player state (stopped/started/paused) as well as health (stale/orphaned from no data)
* Tracks player's reported state, if provided
* Move UserDataSaved heuristics into scrobble method implementation with more descriptive logging
* Add more aggressive UserDataSaved filtering by discarding play if its found in memory tracking, regardless of play date
* Attempt to correct bad UTC offset for second UserDataSaved event
* Refactor candidate/discovered Play maps to use tuple id instead of string so user can be compared later
* Add option to check all discovered platform plays (since jellyfin UserDataSaved doesn't include device id in payload)
* Check for UserDataSaved notification with correct reason as well as sanity check last played date (since jellyfin will save as played regardless of duration played)
* Bypass play tracking
Docusaurus treats the files as assets and forces them to download when relative links. Instead, use external links to github so the user can stay in browser
For jellyfin/plex/tautulli -- `logFilterFailure` option in config allows controlling if and what log LEVEL user/server filter failures are logged. #85
* Updated schema
* Updated examples
* Also added expected/found detailed logging for filters to jellyfin #84
* Split user value on comma to align with what config describes as acceptable (comma-delim)
* Trim found values to fix any whitespace typos
* On disallowed event logging add expected/found values to help with debugging
Fixes#84
When a track contains multiple artist, only the first artist should be send to Last.fm since it doesn't support multiple artists per track.
Previous behaviour was to combine all artists in a single string.
Issue: #81
* Removes need to override poll() in child class and better defines when logic should be performed
* Fixes app crash when spotify credentials are present but invalid #76
Add a scrobbleThresholds object with 'duration' and 'percent' values to all sources and sourceDefaults. If both are defined then first passing condition trigger scrobble.
* duration, in seconds, tests how long MS has been tracking the play (since discovery)
* percent, only used if source provides track duration, tests time MS has been tracking play (since discovery) is some % of track duration
* Remove npm dev dependencies after app install/build
* Delete typescript files after build
* Provide a default config with file logging disabled and copy examples to config folder if empty
* Update winston-duplex to remove hard winston dependency
* Use scoped @foxxmd/winston instead of git commit
* Replace 'winston' with '@foxxmd/winston' in application
* Log warnings when any specific mpris info for a player isn't able to be parsed rather than failing entire poll attempt
* Fallback to undefined tracking progress if position cannot be parsed
* Add play object data "meta" with a brainz property and common mbid tags
* Parse MBID data from jellyfin webhook
* Add MBID data from play object to 'additional_info' when scrobbling to listenbrainz
Closes#72
don't want to assume we won't find an existing just because there are no scrobbles from the client itself -- we may have found it from recent recorded scrobbles sent
* The response track data is different than submission data so make sure we are accurately parsing that data to PlayObjects
* Add separate method for getting listens as recently played (play objects) and refactor getUserListens to return raw payload
* use child loggers instead of creating new loggers for each part of the app
* cleans up and automates nested labels in log output
* fixes "event emitters may be leaky" node warning
* add ability to specify log levels per output type (console, ui, file) using aioconfig
* improve UI logging experience
* use more color-blind friendly colors (borrowed from context-mod)
* add verbose log level output
* increase log buffer and store all logs, then filter at render time, instead of discarding based on min log level so that we always have logs to display
* fix level selection in ui to work for all levels
* Add a 'recentDiscoveredPlays' class property for abstract source
* recentDiscoveredPlays is platform aware and configured based on source 'multiPlatform' class property
* Consolidate scrobble behavior into abstract class
* Check for existing discovered track before scrobbling
* Add to discovered if not existing and increment discovered tracks counter
* Decouple sources from clients by using event emitter when a newly discovered track should be scrobbled
* Use discovered tracks for UI display instead of api data
Since spotify now uses currently playing with memory we need to poll more aggressively to make sure we catch tracks as they are played. Additionally need to decrease max polling interval so as not to miss activity.
* Use recently-played to scrobble backlogged tracks
* Use currently-playing or current-state to implement spotify as a memory source in order to track plays made from spotify connect devices
* HTTP Status as primary indicator with messages in json response
* No parameters in request aggregates all client/source statuses. Otherwise can use 'type' or 'name' parameters to restrict client/sources to aggregate
* Refactor initial connection info to be more verbose and only log once
* Refactor valid event check logging order of operations and make logging more verbose
In order to make basic troubleshooting easier all ingress-based sources (plex, tautulli, jellyfin) now log initial connections and basic request validation checks:
* Request logging includes remote address and user agent
* Log initial requests before any middleware/body parsing
* Include method for checking if request is valid at a low-level (IE checking request verb)
* Include source-specific methods for checking if request payload is valid
* Output to commonjs using TS (remove type:module from package.json)
* Update tsconfig config
* Downgrade formidable to fix ES module import error
* Remove js assertions and use separate json schema for aio client/source validation to keep MS log output cleaner
Although it was nice to include the option no one is providing their own tokens from a separate flow completed elsewhere. It simplifies readability and usage in MS to remove these and always generate out own.
* Update to alpine 3.17 and node 18 to match the project
* Use LSIO base to enable ui/guid usage through env
* Use build stages to reduce image size and speed up build
* Refactor build to use typescript
* Simplify config directory by using root-level /config as default
* Add bash script on startup to copy example configs if a new config folder is detected
MS incorrectly chooses the first unnamed source (based on built source order) when handling spotify callback. Fixed by specify source type when retrieving source. Fixes#63
* Refactor init state to have an interim value and use getter/setter to control it
* Replace usage of 'ready' class field on maloja client with serverIsHealthy (make it client specific)
* Add some missing functions in abstract client class
* If client is not already initialized (or trying to initialize) then try to initialize client on scrobble attempt, maybe fixes#44
* Replace multer with formidable as it is more up-to-date and has more granular options for tracking progress and parsing data
* Add debug logging for the lifecycle of a plex request
* throws errors if any part of plex request lifecycle is unexpected
* Use server version to determine shape of scrobble data when formatting play object
* Refactor adding new scrobble to recent list to use correct shape based on server version
* Use rebuild status and health reported from maloja api endpoint to determine if server is ready to scrobble/query
* Use readiness as prerequisite for auth test (can be removed if krateng/maloja#92 is merged)
Default state is calculated from initialized and auth status. Allows an additional check for client readiness independent of communication (initialized) status and auth status.
* Use ignored count to determine if warning is logged
* Let user know if no error message included
* Include link to error code reference in last.fm api docs
* Emit individual log statements in winston formatter
* Emit different event when logs should be cleared
* Use window global to keep track of sort order when appending on emit
* Break up into different functions and add properties to concrete classes signalling its capabilities
* More detailed logging based on which fails/succeeds
* Simplify client/source init/auth step (no more need for case switch specifics other than creating object)
* Instead of stopping the whole application for one misconfigured config just log the issue and continue with any valid/working configs.
* Add better logging for type.json issues (add index)
* Add deprecation warnings for single-user mode structures in type.json (to be removed in 0.4)
* Instead of stopping the whole application for one misconfigured config just log the issue and continue with any valid/working configs.
* Add better logging for type.json issues (add index)
* Add deprecation warnings for single-user mode structures in type.json (to be removed in 0.4)
* Should stabilize subsonic as a source and prevents duplicate scrobbles
* Reduce polling interval to 10 seconds for more accurate recently played data and restrict max backoff time to 30 seconds
If sources don't support a scrobble action or don't return a sane "recently played" data source we need to keep track of these plays ourselves. Do it in memory based on currently playing and return that as source "source of truth" for recently played.
* Include some metadata (mbid, url, nowplaying)
* Check for more artist properties since responses seem inconsistent now
* Check for timestamp to handle nowplaying use case
* Handle invalid scrobbles from last.fm on recents refresh WRT nowplaying and missing timestamps #22
* Hopefully can get artist from webhook payload in the future so we can actually use this
* Add basic instructions for using Webhook plugin in configuration docs
* Hierarchical retries and delay options for sources and clients (override general config => individual config)
* Logging for retry attempts
* Respect Retry-After header on responses if present
* Pass config dir to client handler
* Rename auth routes to be source/client specific
* Pass client info to status page to enable displaying client stats and actions (auth)
* Handle auth callback from lastfm
* Refactor scrobble clients to be more granular on error handling for scrobble call
* use initialized param on clients to additionally check if they should be used
* Pass config dir to constructor so we can use it any in the class (for lastfm)
* Use error property to determine if we she keep trying to scrobble plays after caught error
* Authentication is user-interaction required with saved session file
* Use initialized to signal auth is done and client is ready to scrobble/get tracks
* Add some retry attempts based on error returned from api
* Use plain ol async because it gets the job done and i don't need no fancy generator/yield. Can wait until loop is done to detect signal to stop polling
* Refactor polling into AbstractSource, remove duplicated code for Spotify/Subsonic sources
* Add 'canPoll' property to Sources
* Stagger polling invocation on app start so log messages don't get jumbled
* Refactor auth/poll/recent endpoints to be generic (based on source properties)
* Refactor status page to show functionality based on canAuth/canPoll rather than type
* Refactor examples to all be multi-user structured
* Move configuration into its own file
* Separate config approaches into env/json approaches with guidance on which one to use
* Add more comments to example json
* Simplify main readme and provide a more opinionated, minimal example there
* Refactor comparison methodology to use weighted scoring system (psuedo-fuzzy) instead of all-or-nothing if statements
* Use reference scrobbles (playObjs we have submitted while app is running) to check for existing
* Make title cleaning more robust by removing parenthesis and all "feat" strings
* Add artists matching
* Rewrite debugging info to show score/breakdowns and add more granular debugging options (show on match/no match)
* Add scrobbledPlayObjs to so we can keep track of formatted playObjs we *know* we have scrobbled (for more accurate comparisons elsewhere)
* Filter scrobbledPlayObjs based on oldest returned recent scrobble from client
* Add instanced generic formatPlayObj for use in other methods
* Add generic source title cleaning function
* Add generic function to search for scrobbled playObj from known submitted playObjs
* Refactor source building into a new class to hold all sources
* ScrobbleSources supports single and multi user config variant parsing (based on client parsing)
* Refactor client/source abstract classes to accept a type and name -- logger is based off of both
* Refactor config file parsing into ScrobbleSources
* Refactor status page to use array of source data instead of hardcoding
* Status page now uses flex css to layout sources instead of hardcoding
* Refactor spotify endpoints to require a name from querystring in order to determine which source to work on (middleware with this)
* Refactor ScrobbleClients to use optional scrobble filter name, passed by configured source
* Implement library and server configuration params for plex/tautulli to help with multi-user
* Make library, server, and user checks on plex/tautulli case insensitive and include more debug information
* Consolidate validEvent checks for plex/tautulli
* Add sane defaults for single-user mode client configs (naming)
* Refactor client building into a two-stage process of 1) parsing and checking for valid config 2) validating config per client
* Refactor client ENVs config into separate config (don't mix/overwrite env and json configs)
* Check for unique names
* Implement more descriptive and thorough config structure validation including hinting at config source (location)
Relates to #13
* Strip source track of feat and [artist] since Maloja does this as well
* Compare position-agnostic and de-duped tokens from source/scrobble titles for a better match
* Refactor playObj artist (string) => artists (array) so we have more info to work with and less ambiguity in naming
* Use forward slash as artist deliminator for Maloja scrobbling because it parses better
* Track building the artists uses forward slash for less ambiguity as well
* Implement func for finding longest string length (for use with padding plays)
* Implement func for truncating string based on a fixed length (for use with padding plays)
* Implement play string building transformers for all parts of the string for finer control
* Implement "X from now" time formatting
* Move all spotifyApi invocation into a wrapped function that handles token refresh so we can just call the api from anywhere without having to worry about re-authenticating
* Implement "get recent played" function using new wrapper and also support formatting results to playObjs before returning
* add open.spotify.com web url to playObj meta if it exists
* Fix an empty maloja history always causing oldest scrobble to now() time, preventing time frame from ever being valid
* Add debug logging to timeframe and existing scrobble checks for new tracks
* Implement api wrapping for maloja and handle formatting error
* Implement testing maloja connection to make sure configuration is valid (check server info and test endpoint)
* Provide better defaults for maloja scrobbles list when empty (maybe fixes#5)
* Better formatting for maloja scrobble api calls
* Better handling of maloja and spotify configuration issues during initialization (And logging for it)
* Move winston format combining and final formatter function into utils
* Use individual loggers for every "area" they are needed by using winston.loggers
* Create a default logger with final transport options
* util convenience function for creating a new labelled format combo
* util convenience function for created new logger with labelled formatter
* remove label usage everywhere! so much cleaner
* remove logger passing everywhere and replace with winston get call for logger we want (from util function)
Too much logging since we are now checking tracks on every spotify polling interval. Really only need to log if we decide its *not* a dupe because of track diffs
* Map all tracks from source/clients immediately to playObjs to reduce cognitive load
* Move responsibility for source and "newFromSource" meta from source -> scrobble clients -- to source -> playObj formatting so playObjs hold all their own meta
* Always immediately sort source/client track lists by playDate ascending for less cognitive load
* Relax valid time frame for maloja client to be based on oldest returned scrobble (since we are always returning most recent scrobbles)
* Refactor debugging for already scrobbled so all diff statements are lumped at the end and formatted
* Simplify spotify source fetching since meta is handled by play objects now
* Fix new track recognition for spotify source by using sorted playObjs instead of relying on order spotify returns tracks
Check all returned tracked from recent spotify plays against scrobbles from clients to see if any have not been scrobbled. This helps compensate for when Spotify has backlogged their own recent plays (plays being "added" to recent response long after they were actually played)
* Check time frame for returned scrobbles from client before checking if a source track has been scrobbled to make sure only source tracks played within valid time frame are scrobbled
* Remove debug logging for recent tracks checks for now -- since all tracks are now checked
* Use different verbiage in logging when track is from backlog vs. new from source client
* Show current level/sort/limit for logging on status page
* Render log config values as links and implement endpoint to update them
Now a user can switch to debug level to check more detailed info without restarting the app
label:Please check existing knowledge before opening an issue
options:
- label:I have [checked the FAQ](https://foxxmd.github.io/multi-scrobbler/docs/FAQ/) and [documentation](https://foxxmd.github.io/multi-scrobbler/docs/configuration/)
required:true
- label:I have [searched through existing issues](https://github.com/FoxxMD/multi-scrobbler/issues?q=sort%3Aupdated-desc) and [discussions.](https://github.com/FoxxMD/multi-scrobbler/discussions)
required:true
- type:markdown
attributes:
value:|
If this a how-do-I question, troublehooting, or generally **not** a bug/error with multi-scrobbler please consider [**starting a discussion**](https://github.com/FoxxMD/multi-scrobbler/discussions) instead of opening an issue.
- type:textarea
id:what-happened
attributes:
label:Describe the Bug
description:Describe what you expected to happen and what happened instead. Please provide as much detail as possible.
placeholder:The app should have scrobbled from Spotify but instead it crashed with an error about undefined properties, etc...
validations:
required:true
- type:dropdown
id:platform
attributes:
label:Platform
description:What platform are you running multi-scrobbler on?
options:
- Docker
- Local (NodeJS)
- Flatpak
default:0
validations:
required:true
- type:textarea
attributes:
label:Versions
description:What version of multi-scrobbler and issue-related sources/clients are running?
placeholder:|
- multi-scrobbler: 0.8.3 on docker
- maloja 3.1.4
- jellyfin 10.8.9
validations:
required:true
- type:textarea
id:logs
attributes:
label:Logs
description:|
Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
If possible reproduce the issue with [debug logging ON](https://foxxmd.github.io/multi-scrobbler/docs/FAQ/#turn-on-debug-logging)
render:shell
- type:textarea
attributes:
label:Additional Context
description:Add any other context about the problem here.
description:Suggest a new feature or functionality
title:"feat: "
labels:["enhancement"]
body:
- type:checkboxes
attributes:
label:Please check existing knowledge before opening an issue
options:
- label:I have checked the [documentation](https://foxxmd.github.io/multi-scrobbler/docs/configuration/)
required:true
- label:I have [searched through existing issues](https://github.com/FoxxMD/multi-scrobbler/issues?q=sort%3Aupdated-desc) and [discussions.](https://github.com/FoxxMD/multi-scrobbler/discussions)
required:true
- type:textarea
attributes:
label:Describe the new functionality
description:Please describe the feature or functionality you want multi-scrobbler to have. Use as much detail as possible about the exact behavior you wish to see.
validations:
required:true
- type:textarea
attributes:
label:Configuration Example
description:|
If the functionality involves changes to configuration files please provide an example here
render:json5
- type:checkboxes
attributes:
label:Will you help implement it?
options:
- label:I am willing to contribute code towards completing this feature
- label:I am willing to help test preview builds for this feature
description:Request a new Source or Client to be implemented
title:"feat: "
labels:["enhancement"]
body:
- type:checkboxes
attributes:
label:Please check existing knowledge before opening an issue
options:
- label:I have checked the [documentation](https://foxxmd.github.io/multi-scrobbler/docs/configuration/)
required:true
- label:I have [searched through existing issues](https://github.com/FoxxMD/multi-scrobbler/issues?q=sort%3Aupdated-desc) and [discussions.](https://github.com/FoxxMD/multi-scrobbler/discussions)
required:true
- type:dropdown
attributes:
label:What Type?
description:Is this a Source or a Client?
options:
- Source
- Client
default:1
validations:
required:true
- type:input
attributes:
label:Name
description:"What is the name of the new service/software?"
placeholder:"My Cool Player"
validations:
required:true
- type:input
attributes:
label:Website
description:"Website for the service/software. Preferably a github repo or other source-code homepage"
validations:
required:true
- type:dropdown
attributes:
label:API Available?
description:Does the service have an API or other protocol that multi-scrobbler can use to communicate? **Without this the likelyhood of implementation is very low.**
options:
- 'Yes'
- 'No'
default:1
validations:
required:true
- type:dropdown
attributes:
label:Free To Use?
description:Is this service/software freely available or does it require purchasing/subscription?
options:
- 'Yes'
- 'No'
default:1
validations:
required:true
- type:dropdown
attributes:
label:Willing to finance development?
description:If the software is **non-free** are you will to provide a paid account/subscription to multi-scrobbler developer's in order to facilitate development/testing?
options:
- 'Yes'
- 'No'
validations:
required:false
- type:checkboxes
attributes:
label:Will you help implement it?
options:
- label:I am willing to contribute code towards completing this feature
- label:I am willing to help test preview builds for this feature
- type:textarea
attributes:
label:Additional Context
description:Add any other context about the new Source/Client here
Need to have credentials in a [`.secrets` file](https://nektosact.com/usage/index.html#secrets), copy and rename [`.secrets.example`](./secrets.example) to `.secrets`, then fill out blank fields. Required for docker/metadata-action to read...something. Fails with `Parameter token or opts.auth is required` if they are not supplied.
An ENV file can also be made by copy and renaming [`.env.example`](./env.example). Set `NO_DOCKER_BUILD=true` if you only want to test APP_VERSION and docker tags output.
If running a full docker build for multi-runner workflows you will need to create an [artifact server](https://github.com/nektos/act/issues/329#issuecomment-1905955589) for ACT to work:
# Based on https://docs.docker.com/build/ci/github-actions/multi-platform/#with-bake (https://github.com/docker/build-push-action/issues/671#issuecomment-1609106171)
# with example from here https://github.com/crazy-max/docker-linguist/blob/master/.github/workflows/build.yml that uses caching
#
# may have to use this to enable pushing to multiple registries? https://github.com/docker/build-push-action/discussions/1067#discussioncomment-8677682
A single-user, javascript app to scrobble your recent plays to [Maloja](https://github.com/krateng/maloja) (and other clients, eventually)
* Displays running status and buffered log through web server
* Spotify - Authorize your app through the web server
* Spotify - Persists obtained credentials to file
* Spotify - Automatically refreshes authorization for unattended use
* Spotify - Implements back off behavior if no listening activity is detected after an interval (after 10 minutes of idle it will back off to a maximum of 5 minutes between checks)
*[Tautulli](https://tautulli.com) - Scrobble endpoint using notification agents
* [Plex](https://plex.tv) - Scrobble endpoint using [Webhooks](https://support.plex.tv/articles/115002267687-webhooks)
A javascript app to scrobble music you listened to, to [Maloja](https://github.com/krateng/maloja), [Last.fm](https://www.last.fm), and [ListenBrainz](https://listenbrainz.org)
* [Plex](https://foxxmd.github.io/multi-scrobbler/docs/configuration#plex) or [Tautulli](https://foxxmd.github.io/multi-scrobbler/docs/configuration#tautulli)
* [Subsonic-compatible APIs](https://foxxmd.github.io/multi-scrobbler/docs/configuration#subsonic) (like [Airsonic](https://airsonic.github.io/) and [Navidrome](https://www.navidrome.org/))
* Monitor status of Sources and Clients using [webhooks (Gotify, Ntfy, Apprise)](https://foxxmd.github.io/multi-scrobbler/docs/configuration#webhook-configurations) or [healthcheck endpoint](https://foxxmd.github.io/multi-scrobbler/docs/configuration#health-endpoint)
* Supports configuring for single or multiple users (scrobbling for your friends and family!)
* Web server interface for stats, basic control, and detailed logs
* Graceful network and client failure handling (queued scrobbles that auto-retry)
* Smart handling of credentials (persistent, authorization through app)
* Easy configuration through ENVs or JSON
* Modify data before scrobbling with [regular expression or search patterns](https://foxxmd.github.io/multi-scrobbler/docs/transforms)
* Install using [Docker images for x86/ARM](https://foxxmd.github.io/multi-scrobbler/docs/installation#docker), [flatpak](https://foxxmd.github.io/multi-scrobbler/docs/installation#flatpak), or [locally with NodeJS](https://foxxmd.github.io/multi-scrobbler/docs/installation#nodejs)
**Why should I use this over a browser extension and/or mobile app scrobbler?**
* **Platform independent** -- Because multi-scrobbler communicates directly with service APIs it will scrobble everything you play regardless of where you play it. No more need for apps on every platform you use!
* **Open-source** -- Get peace of mind knowing exactly how your personal data is being handled.
* **Track your activity regardless of where you listen** -- Scrobble from many Sources to one Client with ease and without duplicating tracks.
* **Manage scrobbling for others** -- Scrobble for your friends and family without any setup on their part. Easily silo sources to specific clients to keep plays separate.
**But I already scrobble my music to Last.fm/ListenBrainz, is multi-scrobbler for me?**
Yes! You can use [Last.fm as a **Source**](https://foxxmd.github.io/multi-scrobbler/docs/configuration#lastfm-source) or [Listenbrainz as a **Source**](https://foxxmd.github.io/multi-scrobbler/docs/configuration#listenbrainz-source) to forward scrobbles from your profile to any other Client! That way you can keep your current scrobble setup as-is but still get the benefit of capturing your data to a self-hosted location.
## How Does multi-scrobbler (MS) Work?
You set up configurations for one or more **Sources** and one or more **Clients**. MS monitors all of your configured **Sources**. When new tracks are played by a Source it grabs that information and then sends it (scrobbles it) to all **Clients** that Source is configured to scrobble to.
### Source
A **Source** is a data source that contains information about tracks you are playing like a music player or platform. Examples are **Spotify, Jellyfin, Plex, Youtube Music, Airsonic**, etc...
Source configurations consist of:
* A friendly name.
* Any data needed to communicate or authenticate with the Source.
* An optional list of Client names that the Source should scrobble to. If omitted the Source also scrobbles to all configured Clients.
### Client
A **Client** is an application that stores the historical information about what songs you have played (scrobbles). Examples are **Maloja, Last.fm, Listenbrainz**...
Client configurations consist of:
* A friendly name.
* Any data needed to communicate or authenticate with the Client.
## Quick Start
[See the **Quick Start Guide**](https://foxxmd.github.io/multi-scrobbler/docs/quickstart)
## Installation
[See the **Installation** documentation](https://foxxmd.github.io/multi-scrobbler/docs/installation)
### Locally
## Configuration
Clone this repository somewhere and then install from the working directory
All configuration is done through json files or environment variables. Reference the [examples in the config folder](config) more detailed explanations and structure.
**A property from a json config will override the corresponding environmental variable.**
### App (General)
[JSON config example](config/config.json.example)
These environmental variables do not have a config file equivalent (to make Docker configuration easier)
| `CONFIG_DIR` | - | `CWD/config` | Directory to look for all other configuration files |
| `LOG_PATH` | - | `CWD/logs` | If `false` no logs will be written. If `string` will be the directory logs are written to |
| `PORT` | - | 9078 | Port to run web server on |
**The app must have permission to write to `CONFIG_DIR` in order to store the current spotify access token.**
### Sources (Where Music Plays Come From)
#### Spotify
To access your Spotify history you must [register an application](https://developer.spotify.com/dashboard) to get a Client ID/Secret. Make sure to also whitelist your redirect URI in the application settings.
| PLEX_USER | - | | The username of the user to scrobble tracks for. No user specified means all tracks by all users will be scrobbled. |
#### [Tautulli](https://tautulli.com)
Check the [instructions](docs/plex.md) on how to setup a notification agent to scrobble your plays.
**Environmental variables and config file is the same as Plex**
### Scrobble Clients
At least one client (the only one right now...) must be setup in order for the app to work. Client configurations can alternatively be configured in the main [`config.json`](https://github.com/FoxxMD/multi-scrobbler/blob/master/config/config.json.example) configuration
| `MALOJA_URL` | Yes | | Base URL of your installation |
| `MALOJA_API_KEY` | Yes | | Api Key |
[See the **Configuration** documentation](https://foxxmd.github.io/multi-scrobbler/docs/configuration)
## Usage
A status page with statistics and recent logs can found at
A status page with statistics, recent logs, and some runtime configuration options can be found at
```
https://localhost:9078
http://localhost:9078
```
Output is also provided to stdout/stderr as well as file if specified in configuration.
On first startup you may need to authorize Spotify by visiting the callback URL (which can also be accessed from the status page)
On first startup you may need to authorize Spotify and/or Last.fm by visiting the callback URL (which can also be accessed from the status page). Visit the status page above to find the applicable link to trigger this.
```
https://localhost:9078/authSpotify
```
## Help/FAQ
### Running Directly
Having issues with connections or configuration? Check the [FAQ](https://foxxmd.github.io/multi-scrobbler/docs/FAQ) before creating an issue!
this.logger.debug(`Scrobble with same name found and the play (start time) vs. scrobble time diff was smaller than 10 seconds`,{label:this.name});
returntrue;
}
this.logger.debug(`Scrobble with same name found but the start/finish times vs scrobble time diffs were too large to consider dups (Start Diff ${scrobblePlayStartDiff.toFixed(0)}s) (End Diff ${scrobblePlayDiff.toFixed(0)}s)`,{label:this.name});
These are **example configurations** for all Source/Client types and AIO config.
These can be used as-is by renaming them to `.json` and filling or replacing sample data.
For docker installations these examples are copied to your configuration directory on first-time use.
These are **NOT** exhaustive examples. You should consult the **configuration** documentation and the **schema explorer links** for each source/config type to see a complete list of options and descriptions for all properties.
- TZ=Etc/GMT# Specify timezone from TZ Database name found here https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
#- SPOTIFY_CLIENT_ID=
#- SPOTIFY_CLIENT_SECRET=
#- SPOTIFY_REDIRECT_URI=http://multi-scrobbler-host-IP:9078/callback ## Need to be whitelisted in Spotify dashboard and is used for creating the connection first time
#- MALOJA_URL=http://maloja:42010
#- MALOJA_API_KEY=
#- PUID=1000 # required if running docker on linux host, see main README Docker setup instructions
#- PGID=1000 # required if running docker on linux host, see main README Docker setup instructions
# set if using a source/client with redirect URI that you have not explicitly set and MS is NOT running on the same machine that you will view the dashboard from
# EX: You will view MS dashboard at 'http://192.168.0.101:9078' -> set BASE_URL=http://192.168.0.101:9078
#- BASE_URL=http://MyHostIP:9078
volumes:
- "./config:/config"
#networks:
# - (optional to add container to the same bridge network that maloja is inside to be able to use docker internal networking & dns to resolve and connect to maloja URL via http://maloja:port)
ports:
- 9078:9078# first port is the HOST port multi-scrobbler will serve UI and API on
Note: I do not recommend running a container without a `config` volume specified or you will need to reauthorize the app everytime the container is rebuilt.
These sources are **ingress-based** which means that multi-scrobbler waits for the Plex/Tautulli/Jellyfin server (or Webscrobbler extension) to contact multi-scrobbler, as opposed to multi-scrobbler contacting them.
multi-scrobbler will log information about any server that connects to it for these services. In the logs it looks something like this:
```
2023-02-22T10:55:56-05:00 info : [Ingress - Plex ] Received request from a new remote address: ::ffff:192.168.0.140 (UA: PlexMediaServer/1.24.5.5173-8dcc73a59)
2023-02-22T10:55:56-05:00 info : [Ingress - Plex ] ::ffff:192.168.0.140 (UA: PlexMediaServer/1.24.5.5173-8dcc73a59) Received valid data from server examplePlex for the first time.
2023-02-22T10:55:56-05:00 warn : [Plex Request ] Received valid Plex webhook payload but no Plex sources are configured
```
It also logs if a server tries to connect to a URL that it does not recognize:
```
2023-02-22T11:16:12-05:00 debug : [App ] Server received POST request from ::ffff:192.168.0.140 (UA: PlexMediaServer/1.24.5.5173-8dcc73a59) to unknown route: /plkex
```
**So, if you do not see either of these in your logs then Plex/Tautulli/Jellyfin/Webscrobbler is not able to connect to your multi-scrobbler instance at all.**
This is not something multi-scrobbler can fix and means you have an issue in your network.
#### Troubleshooting
Check or try all these steps before submitting an issue:
##### Turn on Debug Logging
First, turn on **debug** logging for multi-scrobbler by setting the environmental variable `LOG_LEVEL=debug`:
* using node `LOG_LEVEL=debug ... node src/index.js`
* using docker `docker run -e LOG_LEVEL=debug ... foxxmd/multi-scrobbler`
Check the output for any additional information.
##### Check Host name and URL
The URLs examples in the [configuration](configuration/configuration.mdx) documentation assume you are running Plex/Tautulli/Jellyfin/Webscrobbler on the same server as multi-scrobbler. If these are not the same machine then you need to determine the IP address or domain name that multi-scrobbler is reachable at and use that instead of `localhost` when configuring these sources. **This is likely the same host name that you would use to access the web interface for multi-scrobbler.**
EX `http://localhost:9078/plex` -> `http://192.168.0.140:9078/plex`
##### Check Firewall and Port Forwarding
If the machine multi-scrobbler is running on has a firewall ensure that port **9078** is open. Or if it is in another network entirely make sure your router is forwarding this port and it is open to the correct machine.
##### Check Source Service Logs
Plex/Tautulli/Jellyfin all have logs that will log if they cannot connect to multi-scrobbler. Check these for further information.
###### Plex
Settings -> Manage -> Console
###### Tautulli
Check the command-line output of the application or docker logs.
###### Jellyfin
Administration -> Dashboard -> Advanced -> Logs
###### Webscrobbler
See [Debugging the extension](https://github.com/web-scrobbler/web-scrobbler/wiki/Debug-the-extension) to get logs which should have information about failed requests.
### Jellyfin has warnings about undefined or missing data
Make sure you have
* [Configured the webhook plugin correctly](configuration/configuration.mdx#jellyfin)
* Checked the **Send All Properties(ignores template)** option in the webhook settings and **Saved**
multi-scrobbler is known to work on Jellyfin `10.8.9` with Webhook version `11.0.0.0`.
You can verify the payload sent from the webhook by modifying your jellyfin configuration to include `logPayload: true` which will output the raw payload to DEBUG level logging:
```json
[
{
"name":"MyJellyfin",
"clients":[],
"data":{
"users":["FoxxMD"]
},
"options":{
"logPayload":true
}
}
]
```
If your issue persists and you open an Issue for it please include the raw payload logs in your report.
### Jellyfin has warnings about missing headers
If you experience issues trying to scrobble with Jellyfin and find this in your MS logs
```
[API] Jellyfin is not sending a request with valid headers...
```
A workaround that may fix this:
* In Webhook settings:
* [In the webhook you have already configured...](configuration/configuration.mdx#jellyfin)
* Add Request Header...
* **Key:** `Content-Type`
* **Value:** `application/json`
* Then Save
### Spotify/Deezer/LastFM won't authenticate
Ensure any **client id** or **secrets** are correct in your configuration.
The callback/redirect URL for these services must be:
* the same address you would use to access the multi-scrobbler web interface
* the web-interface must be accessible from the browser you are completing authentication from.
If multi-scrobbler is not running on the same machine your browser is on then the default/example addresses (`http://localhost...`) **will not work.** You must determine the address you can reach the web interface at (such as `http://192.168.0.140:9078`) then use that in place of `localhost` in the callback URLs.
EX `http://localhost:9078/lastfm/callback` -> `http://192.168.0.220:9078/lastfm/callback`
### Deezer is not working
Deezer has discontinued support for their API and the Deezer Source is now [**deprecated.**](configuration/configuration.mdx#deezer) See [this issue for more discussion.](https://github.com/FoxxMD/multi-scrobbler/issues/175#issuecomment-2296776625)
### Youtube Music fails after some time
The Youtube Music library relies on scraping the YTM site (pretending to be a browser) by using cookies/auth from your actual browser. It does its best to keep these up to date but since this is not an official way to access the service YTM may invalidate your access _to the authenticated session_ at any time. How this is triggered is unknown and not something multi-scrobbler can control.
If you see errors in multi-scrobbler for YTM that contain **401** or **403** like
```
Error: Could not send the specified request to browse. Status code: 401
```
then YTM has invalidated your access. [Follow the YTM instructions to retrieve a new set of cookies for multi-scrobbler](configuration/configuration.mdx#youtube-music) and then restart MS to potentially resolve the problem. See [this issue](https://github.com/FoxxMD/multi-scrobbler/issues/158) for further discussion of the problem.
## Configuration Issues
### Config could not be parsed
If you see something like this in your logs:
```
2023-02-19T10:05:42-06:00 warn : [App] App config file exists but could not be parsed!
2023-02-19T10:05:42-06:00 error : [App] Exited with uncaught error
2023-02-19T10:05:42-06:00 error : [App] Error: config.json could not be parsed
```
It means the JSON in your configuration file is not valid. Copy and paste your configuration into a site like [JSONLint](https://jsonlint.com/) to find out where errors you have and fix them.
## Scrobbling Issues
### Last.fm does not scrobble tracks with multiple artists correctly
This is a limitation of the [Last.fm API](https://www.last.fm/api/show/track.scrobble) where the **artist** field is only one string and Last.fm does not recognize (play well) with "combined" artists.
Multi-scrobbler works the same was the official Spotify-Last.fm integration works -- it only scrobbles the **first** artist on a multi-artist track.
### Jellyfin does not scrobble tracks with multiple artists correctly
This is a limitation caused by the [Jellyfin webhook plugin](https://github.com/FoxxMD/multi-scrobbler/issues/70#issuecomment-1443804712) only sending the first artist to multi-scrobbler. This issues needs to be [fixed upstream on the Jellyfin webhook repository.](https://github.com/jellyfin/jellyfin-plugin-webhook/issues/166)
### Google Cast track information is missing/incorrect or MS player has weird times
The Google Cast integration relies on a few common fields in the data it receives from your casting device. Every platform that can cast (Spotify, Pandora, etc...) *should* use these fields the same but there are slight differences between their implementations that may confuse multi-scrobbler. Specific platforms may also return more information in non-common fields that are undocumented.
To diagnose these issues you [**must enable payload logging**](configuration/configuration.mdx#cast-troubleshooting) for your google cast Source, run MS, and then include logs with this output from that run. Without the raw data logged from your cast device it will be nearly impossible to resolve your issue.
### Google Cast device does not track media
It is likely the app playing on the cast device is incorrectly reporting the media type as **not music**.
MS logs will tell you what type the media is reported as with lines like:
```
My Artist - Example Track has 'unknown' media type and allowUnknownMedia=false, will not track
```
Refer to [Allow Unknown Media Type](configuration/configuration.mdx#allow-unknown-media-type) section to fix this
```
My Artist - Example Track has 'movie' media type so will not track
```
Refer to [Force Media Tracking](configuration/configuration.mdx#forcing-media-tracking) section to fix this
### VLC is not scrobbling fields correctly
Before reporting an issue turn on metadata logging in the MS VLC configuration, [see the VLC documentation.](configuration/configuration.mdx#vlc-information-reporting)
### Youtube Music misses scrobbles
In order for multi-scrobbler to accurately determine if a song has been scrobbled it needs **a source of truth.** For YTM this is a "history" list scraped from the YTM website. Unfortunately, the data in this list can be (often) inconsistent which makes it hard for multi-scrobbler to "trust" that it is correct and determine when/if new scrobbles occur. This inconsistency is not something multi-scrobbler can control -- it is a side-effect of having to use an unofficial method to access YTM (scraping).
In order to compensate for this multi-scrobbler resets when it considers this list the "source of truth" based on if the list changes in an inconsistent way between consecutive checks. New scrobbles can only be detected when this list is "OK" as a source of truth for N+1 checks. Therefore, any new tracks that appear when the list is inconsistent will be ignored.
See [this issue](https://github.com/FoxxMD/multi-scrobbler/issues/156#issuecomment-2312533486) for further discussion and a more detailed explanation of why this is happening and how multi-scrobbler compensates for it.
* You want to scrobble plays for yourself (Foxx), Fred, and Mary
* Each person has their own Maloja server
* Each person has their own Spotify account
* You have your own Airsonic (subsonic) server you to scrobble from
* You have your own Youtube Music account you want to scrobble from
* Mary has her own Last.fm account she also wants to scrobble to
* Fred has his own Spotify application and provides you with just his access and refresh token because he doesn't trust you (wtf Fred)
* Fred has a Plex server and wants to scrobble everything he plays
* Mary uses Fred's Plex server but only wants to scrobble her plays from the `podcast` library
* The three of you have a shared library on Plex called `party` that you only play when you are hanging out. You want plays from that library to be scrobbled to everyone's servers.
* Fred also has his own Jellyfin server and wants to scrobble everything he plays
### All-in-one Config
```json5 title="CONFIG_DIR/config.json"
{
"sourceDefaults": {
"maxPollRetries": 0, // optional, default # of automatic polling restarts on error. can be overridden by property in individual config
"maxRequestRetries": 1, // optional, default # of http request retries a source can make before error is thrown. can be overridden by property in individual config
"retryMultiplier": 1.5, // optional, default retry delay multiplier (retry attempt * multiplier = # of seconds to wait before retrying). can be overridden by property in individual config
},
"clientDefaults": {
"maxRequestRetries": 1, // optional, default # of http request retries a client can make before error is thrown. can be overridden by property in individual config
"retryMultiplier": 1.5, // optional, default retry delay multiplier (retry attempt * multiplier = # of seconds to wait before retrying). can be overridden by property in individual config
},
"sources": [
{
"type": "spotify",
"name": "foxxSpot",
"clients": ["foxxMaloja"],
"data": {
"clientId": "foxxSpotifyAppId",
"clientSecret": "foxxSpotifyAppSecret",
},
"options": {
"maxRequestRetries": 2, // override default max retries because spotify can...spotty
}
},
{
"type": "spotify",
"name": "marySpot",
"clients": ["maryMaloja"],
"data": {
"clientId": "foxxSpotifyAppId", // only need one application, it can be used by all users of this multi-scrobbler instance
"clientSecret": "foxxSpotifyAppSecret",
}
},
{
"type": "spotify",
"name": "fredSpot",
"clients": ["fredMaloja"],
"data": {
"accessToken": "fredsToken",
"refreshToken": "fredsRefreshToken",
"interval": 120, // he also wants a slower check interval because his application already has heavy api usage
}
},
{
"type": "plex",
"name": "fredPlex",
"clients": ["fredMaloja"],
"data": {
"user": ["fred@email.com"]
}
},
{
"type": "plex",
"name": "maryPlex",
"clients": ["maryMaloja"],
"data": {
"user": ["mary@email.com"], // still need to specify mary as user so not all users who play from 'podcasts' get scrobbled
"libraries": ["podcasts"]
}
},
{
"type": "plex",
"name": "partyPlex",
// omitting clients (or making it empty) will make this Source scrobble to all Clients
"data": {
"libraries": ["party"],
}
},
{
"type": "jellyfin",
"name": "FredJelly",
// omitting clients (or making it empty) will make this Source scrobble to all Clients
Multi-scrobbler configs support the ability to modify scrobble data in an automated fashion by matching and replacing strings in **title, artists, and album** at many different times in multi-scrobbler's lifecycle.
### Why?
You may need to "clean up" data from a Source or before sending to a scrobble Client due to any number of reasons:
* ID3 tags in your music collection are dirty or have repeating garbage IE `[YourMusicSource.com] My Artist - My Title`
* A Source's service often incorrectly adds data to some field IE `My Artist - My Title (Album Version)` when the title should just be `My Title`
* An Artist you listen to often is spelled different between a Source and a Client which causes duplicate scrobbles
In any scenario where a repeating pattern can be found in the data it would be nice to be able to fix it before the data gets downstream or to help prevent duplicate scrobbling. Multi-scrobbler can help you do this.
## Overview
### Journey of a Scrobble
First, let's recap the lifecycle of a scrobble in multi-scrobbler:
**Sources** are the beginning of the journey for a **Play** (song you've listened to long enough to be scrobblable)
* A Source finds a new valid **Play**
* The Source **compares** this new Play to all the other Plays it has already seen, if the Play is unique (title/artist/album/listened datetime) then...
* The Source **discovers** the Play, adds it to Plays it has seen already, and broadcasts the Play should be scrobbled to all Clients
Scrobble **Clients** listen for discovered Plays from Sources, then...
* A Client receives a **Play** from a Source
* The Client **compares** this Play to all the other scrobbles it has already seen, if the Play is unique (title/artist/album/listened datetime) then...
* The Client **scrobbles** the Play downstream to the scrobble service and adds it as a Scrobble it has seen already
### Lifecyle Hooks
You'll notice there is a pattern above that looks like this:
* **Before** data is compared
* Data is **compared**
* **After** data is compared
These points, during both Source and Client processes, are when you can hook into the scrobble lifecycle and modify it.
#### TLDR
In more concrete terms this is the structure of hooks within a configuration (can be used in any **Source** or **Client**):
```json5 title="lastfm.json" {10-14}
[
{
"name": "myLastFm",
"enable": true,
"configureAs": "source",
"data": {
// ...
},
"options": {
"playTransform": {
"preCompare": {/* ... */},
"compare": {/* ... */},
"postCompare": {/* ... */}
}
}
}
]
```
##### Hook
For **Sources**:
* `preCompare` - modify Play data immediately when received
* `compare` - temporarily modify Play data when it is being compared to see if Play was already discovered
* `postCompare` - modify Play data before sending to scrobble **Clients**
For **Clients**:
* `preCompare` - modify Play data immediately when received
* `compare` - temporarily modify Play data when it is being compared to see if it was already scrobbled
* `postCompare` - modify Play data before scrobbling it to downstream service and adding to already seen scrobbles
:::tip
Keep in mind that modifying Scrobble/Play data earlier in the lifecycle will affect that data at all times later in the lifecycle.
For example, to modify the track so it's the same anywhere it is processed in multi-scrobbler you only need to modify it in the **Source's** `preCompare` hook because all later processes will receive the data with the modified track.
:::
### Modification Parts
Each [**hook**](#hook) (`preCompare` etc...) is an object that specifies what part of the **Play** to modify:
```json5
{
"title": [/* ... */],
"artists": [/* ... */],
"album": [/* ... */]
}
```
##### Expression
and then a **list** what pattern/replacements (expressions) to use for the modification by using either simple strings or `search-replace` objects:
```json5
[
"badTerm", // remove all instances of 'badTerm'
{
"search": "anotherBadTerm", // and also match all instances of 'anotherBadTerm'
"replace": "goodTerm" // replace with the string 'goodTerm'
}
]
```
Putting it all together:
```json5 title="lastfm.json"
[
{
"name": "myLastFm",
"enable": true,
"configureAs": "source",
"data": {
// ...
},
"options": {
"playTransform": {
"preCompare": {
"title": [
[
"badTerm",
{
"search": "badTerm",
"replace": "goodTerm"
}
]
]
},
}
}
}
]
```
:::note
If the value of the field (title, an artist, album) is an empty string after transforming then the field is **removed.**
:::
:::tip
Modifications can also be applied to **all Sources** or **all Clients** when using the [AIO Config](./configuration.mdx?configType=aio#configuration-types) `config.json` by setting `playTransform` in `sourceDefaults` or `clientDefaults`:
<details>
<summary>Example</summary>
```json5 title="config.json"
{
"sourceDefaults": { // will apply playTransform to all sources
"playTransform": {
"preCompare": {
"title": [
"(Album Version)"
]
}
}
},
"sources": [/* ... */],
"clients": [/* ... */]
}
```
</details>
:::
#### Compare Hook
The `compare` [hook](#hook) is slightly different than `preCompare` and `postCompare`. It consists of an object where you define which side(s) of the comparison should be modified. It also **does not modify downstream data!** Instead, the modifications are made only for use in the comparison.
```json5 title="lastfm.json"
[
{
"name": "myLastFm",
// ...
"options": {
"playTransform": {
"compare": {
"candidate": {/* ... */}, // modify the "new" Play being compared
"existing": {/* ... */}, // modify all "existing" Play/Scrobbles the new Play is being compared against
},
}
}
}
]
```
#### Regular Expressions
In addition to plain strings [expressions](#expression) that are matched and removed you can also use Regular Expressions. Write your regex like you normally would, but as a string, and it'll automatically be parsed:
```json5
[
"/^\(\w+.com)/i", // matches any string that starts with '(YourMusic.com)' and removes it
{
"search": "/^\(\w+.com)/i", // matches any string that starts with '(YourMusic.com)'
"replace": "[MySite.com]" // replace with the string '[MySite.com]'
}
]
```
The `replace` property uses javascript's [`replace()` function and so can use any special string characters.](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace#specifying_a_string_as_the_replacement)
### Conditional Modification
#### "When" Condition
Top-level hooks **and** individual rules also support a `when` key for testing **if they should be run.**
The `when` key is similar to a normal [modification](#modification-parts) except:
* the keys accept a single string instead of an array
* the `when` key data is an array instead of a single object
All parts of an individual `when` clause must test true to "pass" but if **any** `when` clauses pass the hook/rule is processed. Example `when` data:
```json5
{
"when": [
{
"artist": "Elephant Gym", // both of these must match the Play object (AND)
"album": "Dreams" // both of these must match the Play object (AND)
},
// OR
{
"title": "/(Remastered)$/", // both of these must match the Play object (AND)
"album": "Various Artists" // both of these must match the Play object (AND)
}
]
}
```
More succinctly:
* All parts (`artist` `album` `title`) of a `when` are `AND` conditions
* All part-objects in the `when` array are `OR` conditions
<details>
<summary>Example of top-level hook with when condition</summary>
```json5
{
// IF the artist is Elephant Gym
// THEN Run preCompare hook ELSE skip this hook
//
// Run search-replace on album
// Run regex title remove
"sourceDefaults": {
"playTransform": {
"preCompare": {
"when": [
{
"artist": "/Elephant Gym/"
}
],
"album": [
{
"search": "Dreams",
"replace": "夢境"
}
],
"title": ["/\s\-\s滾石40\s滾石撞樂隊\s40團拚經典(.+)$/i"]
},
}
}
}
```
</details>
<details>
<summary>Example of individual rule with when condition</summary>
```json5
{
// Always run preCompare
//
// On search-replace in title...
// IF artist matches "Elephant Gym"
// THEN Run regex search-replace ELSE skip this rule
Top-level hooks can also be an array of hooks. This makes creating multiple scenarios for top-level `when`-gated hooks easier. All hooks in the array will be run (assuming their `when`'s pass, if they exist) and their **input will be the Play object output of the previous hook in the array.**
<details>
<summary>Example</summary>
```json5
{
"sourceDefaults": {
"playTransform": {
"preCompare": [
// first lifecycle hook of preCompare to run
{
"title": [
{
"search": "something",
"replace": "else unique"
}
]
},
// second lifecycle hook of preCompare to run
{
"title": [
{
"search": "else unique",
"replace": "very demure"
}
]
},
]
}
}
}
```
</details>
### Logging
MS can log the output of hook transformations if/when they occur. In the `playTransform` object of a Source/Client config use `log`:
* `"log": true` => Output original play + final transformed output of last hook in the array
* `"log": "all"` => Output original play + final transformed output of **each** hook in the array
```json5
{
"name": "myThing",
"data": {/*...*/},
"options": {
"playTransform": {
"preCompare": {/*...*/},
"log": true
}
}
}
```
## Examples
### Remove phrase from Title in all new Plays
Removes the phrase `(Album Version)` from the Title of a Play
<details>
<summary>Example</summary>
```json5 title="config.json"
{
"sourceDefaults": {
"playTransform": {
"preCompare": {
"title": [
"(Album Version)"
]
}
}
}
}
```
</details>
### Remove all parenthesized content from the end of a title
<details>
<summary>Example</summary>
```json5 title="lastfm.json"
[
{
"name": "myLastFm",
// ...
"options": {
"playTransform": {
"compare": {
"candidate": {
"title": [
"/(\(.+\))\s*$/"
]
},
"existing": {
"title": [
"/(\(.+\))\s*$/"
]
},
},
}
}
}
]
```
</details>
### Rename misspelled artist in all new Plays
<details>
<summary>Example</summary>
```json5 title="config.json"
{
"sourceDefaults": {
"playTransform": {
"preCompare": {
"artists": [
{
"search": "Boz Skaggs",
"replace": "Boz Scaggs"
}
]
}
}
}
}
```
</details>
### Remove "Various Artists" albums in all new Plays
Multi-scrobbler is written entirely in [Typescript](https://www.typescriptlang.org/). It consists of a backend and frontend. The backend handles all Source/Client logic, mounts web server endpoints that listen for Auth callbacks and Source ingress using [expressjs](https://expressjs.com/), and serves the frontend. The frontend is a standalone [Vitejs](https://vitejs.dev/) app that communicates via API to the backend in order to render the dashboard.
## Project Setup
Development requires [Node v18.19.1](https://nodejs.org/en) or higher is installed on your system.
:::tip
When running locally (not with a devcontainer) you can use [nvm](https://github.com/nvm-sh/nvm) to manage the installed node version.
:::
Clone this repository somewhere and then install from the working directory
nvm use # optional, sets correct node version when running without devcontainer
npm install
npm run start
```
### VSCode
This repository contains [workspace settings](https://github.com/FoxxMD/multi-scrobbler/blob/master/.devcontainer) for development with VSCode. These include:
* Run/Debug [Launch configurations](https://code.visualstudio.com/Docs/editor/debugging#_launch-configurations) for the application and tests
* [Devcontainer](https://code.visualstudio.com/docs/devcontainers/containers) for development with all dependencies already installed
* Useful extensions for linting and running tests
To use the Devcontainer simple open the repository in VSCode and "Use Devcontainer" when the notification is presented. `npm install` will be run when a new container is created.
## Common Development
:::info
In this document, when referring to aspects of Sources and Clients that are shared between both, the Source/Client will be referred to as a **Component.**
:::
A Component is composed of two parts:
* Typescript interfaces describing structure of configuration for that Component
* A concrete class inheriting from a common "startup" abstract class that enforces how the Component is built and operates
In both parts Source/Clients share some common properties/behavior before diverging in how they operate.
### Config
The configuration for a Component should always have this minimum shape, enforced respectively by the interfaces [CommonSourceConfig](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/common/infrastructure/config/source/index.ts#L105) and [CommonClientConfig](https://github.com/FoxxMD/multi-scrobbler/blob/ce1c70a4e1e87fb5bea7cca960eaafbd15881a1f/src/backend/common/infrastructure/config/client/index.ts#L68):
```ts
interfaceMyConfig{
name: string
data?: object
options?: object
}
```
*`data` contains data that is required for a Component to operate such as credentials, callback urls, api keys, endpoints, etc...
*`options` are **optional** settings that can be used to fine-tune the usage of the Component but are not required or do not majorly affect behavior. EX additional logging toggles
### Concrete Class
Components inherit from an abstract base class, [`AbstractComponent`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/common/AbstractComponent.ts), that defines different "stages" of how a Component is built and initialized when MS first starts as well as when restarting the Component in the event it stops due to an error/network failure/etc...
#### Stages
Stages below are invoked in the order listed. All stages are asynchronous to allow fetching network requests or reading files.
The stage function (described in each stage below) should return a value or throw:
* return `null` if the stage is not required
* return `true` if the stage succeeded
* return a `string` if the stage succeeded and you wish to append a result to the log output for this stage
* throw an `Exception` if the stage failed for any reason and the Component should not continue to run/start up
##### Stage: Build Data
This stage should be used to validate user configuration, parse any additional data from async sources (file, network), and finalize the shape of any configuration/data needed for the Component to operate.
:::info
Implement [`doBuildInitData`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/common/AbstractComponent.ts#L71) in your child class to invoke this stage.
::::
<details>
<summary>Examples</summary>
* Parse a full URL like `http://SOME_IP:7000/subfolder/api` from user config containing a base url like `data.baseUrl: 'SOME_IP'` and then store this in the class config
* Validate that config `data` contains required properties `user``password``salt`
* Read stored credentials from `${this.configDir}/currentCreds-MySource-${name}.json`;
</details>
##### Stage: Check Connection
This stage is used to validate that MS can communicate with the service the Component is interacting with. This stage is invoked on MS startup as well as any time the Component tries to restart after a failure.
If the Component depends on **ingress** (like Jellyfin/Plex webhook) this stage is not necessary.
:::info
Implement [`doCheckConnection`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/common/AbstractComponent.ts#L103) in your child class to invoke this stage.
::::
<details>
<summary>Examples</summary>
* Make a [`request`](https://nodejs.org/docs/latest-v18.x/api/http.html#httprequesturl-options-callback) to the service's server to ensure it is accessible
* Open a websocket connection and check for a ping-pong
</details>
##### Stage: Test Auth
MS determines if Auth is required for a Component based on two class properties. You should set these properties during `constructor` initialization for your Component class:
*`requiresAuth` - (default `false`) Set to `true` if MS should check/test Auth for this Component
*`requiresAuthInteraction` - (default `false`) Set to `true` if user interaction is required to complete auth IE user needs to visit a callback URL
If the Component requires authentication in order to communicate with a service then any required data should be built in this stage and a request made to the service to ensure the authentication data is valid.
This stage should return:
*`true` if auth succeeded
*`false` if auth failed without unexpected errors
* IE the authentication data is not valid and requires user interaction to resolve the failure
* throw an exception if network failure or unexpected error occurred
You _should_ attempt to re-authenticate, if possible. Only throw an exception or return `false` if there is no way to recover from an authentication failure.
:::info
Implement [`doAuthentication`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/common/AbstractComponent.ts#L111) in your child class to invoke this stage.
::::
<details>
<summary>Examples</summary>
* Generate a Bearer Token for Basic Auth from user/password given in config and store in class properties
* Make a request to a known endpoint with Authorization token from read credentials file to see if succeeds or returns 403
* Catch a 403 and attempt to reauthenticate at an auth endpoint with user/password given in config
</details>
### Play Object
The **PlayObject** is the standard data structure MS uses to store listen (track) information and data required for scrobbling. It consists of:
* Track Data -- a standard format for storing track, artists, album, track duration, the date the track was played at, etc...
* Listen Metadata -- Optional but useful data related to the specific play or specifics about the Source/Client context for this play such as
* Platform specific ID, web URL to track, device/user ID that played this track, etc...
Both Sources and Clients use the **PlayObject** interface. When a Component receives track info from its corresponding service it must transform this data into a PlayObject before it can be interacted with.
For more refer to the TS documentation for `PlayObject` or [`AmbPlayObject`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/core/Atomic.ts#L141) in your project
## Creating Clients and Sources
* [Source Development and Tutorial](dev-source.md)
* [Client Development and Tutorial](dev-client.md)
This document will provide a step-by-step guide for creating a (trivial) new Source in MS alongside describing what aspects of the Source need to be implemented based on the service you use. Before using this document you should review [Common Development](dev-common.md#common-development).
## Scenario
You are the developer of a fancy, new self-hosted web-based media player called **Cool Player.** Cool Player has a slick interface and many bells and whistles, but most importantly it has an API. The API:
* Has an unauthenticated health endpoint at `/api/health` that returns `200` if the service is running properly
* Has authenticated endpoints that require a user-generated token in the header `Authorization MY_TOKEN`
* Has a `/api/recent` endpoint that lists recently played tracks with a timestamp
* Has a `/api/now-playing` endpoint that returns information about the state of the player like current track, player position in the track, etc...
* Cool Player is by default accessed on port `6969`
* Your personal instance of Cool Player is hosted at `http://192.168.0.100:6969` and the api is accessed at `http://192.168.0.100:6969/api`
Because there is an API that MS can actively read this will be a **polling** Source where MS sends requests to Cool Player to get scrobble information -- as opposed to an **ingress** Source like Jellyfin/Plex that uses webhooks from the service to send data to MS.
## Minimal Implementation
### Define and Implement Config
We will create a new config interface for Cool Player using the [Common Config](dev-common.md#config) and tell MS it is a valid config that can be used.
* implements required methods to current player state and/or now playing track
### Create CoolPlayer Source
First we create a new Source called `CoolPlayerSource` and setup our constructor to accept the config and [specify Auth behavior.](dev-common.md#stage-test-auth)
// but does not require user interaction for auth to work
this.requiresAuthInteraction = false;
// tells MS this is a Source that can be activity monitored through API
this.canPoll = true;
}
}
```
### Initialize Source from Config
When MS starts it reads all configs and determines which Source to build based on the configs found. We need to tell it to build a `CoolPlayerSource` when a `coolplayer` config type is found.
We modify `ScrobbleSources.ts` to add `CoolPlayerSource` as an option:
newSource = await new CoolPlayerSource(name, compositeConfig as CoolPlayerSourceConfig, internal, this.emitter);
break;
// highlight-end
default:
break;
}
}
}
```
### Implement Play Object Transform
Now we will create a static function that is used to take the track data returned from Cool Player's API and return a standard [`PlayObject`.](dev-common.md#play-object)
Next we will implement the [Stages](dev-common.md#stages) required to get CoolPlayerSource running.
#### Build Data
First we implement the [Build Data Stage](dev-common.md#stage-build-data). We will check that the `token` and `baseUrl` properties have been provided by the user. Additionally, we will parse the baseUrl and add default ports/prefix.
// if Cool Player returned an error as json we could parse it from error body and throw new Error with the message
throw e;
}
}
}
```
### Implement Polling
The majority of Sources MS monitors primarily operate as a source of truth for a **music player** rather than a **played music history.** Only Listenbrainz and Last.fm operate as a source of music history.
To this end, MS implements a [state machine](https://www.freecodecamp.org/news/state-machines-basics-of-computer-science-d42855debc66/) that emulates the behavior of a music player in order to keep track of when a song you are listening to should be scrobbled. It does this by monitoring the "currently playing" track reported by a Source's service, with varying degrees of accuracy depending on what information is returned from the service. The state machine is implemented in `MemorySource` which our `CoolPlayerSource` inherits from.
For a polling Source to work properly we need to implement a function, [`getRecentlyPlayed`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/sources/AbstractSource.ts#L92), that returns PlayObjects that are "newly" played. These are then checked against previously "discovered" plays and their timestamp to determine if they should be surfaced to Clients to scrobble.
To take advantage of the `MemorySource` state machine we will additionally use [`processRecentPlays`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/sources/MemorySource.ts#L113) from `MemorySource` inside `getRecentlyPlayed`. We pass track and/or player state returned from the Source service to `processRecentPlayers`. It then takes care of deriving Source player state based on how this data changes over time. The advantage to using `processRecentPlays` is that our Source service does not necessarily need to pass any player information -- as long as the track info has a **duration** we can more-or-less determine if it has been played long enough to scrobble.
playerState, // 'playing' or 'stopped' or 'paused'...
position, // number of seconds into the track IE at position 48 -> ( 0:48/3:56 in player UI )
play: { /* track data */}
}
} = resp;
// transform into standard player state data
const playerData: PlayerStateData = {
platformId: SINGLE_USER_PLATFORM_ID,
play: CoolPlayerSource.formatPlayObj(play),
position
};
// if Cool Player does return player state we can also push a regular PlayObject to this list
plays.push(playerData);
} catch (e) {
throw e;
}
// process player state through state machine
// if the track changes or player state changes
// and currently played track has been listened to long enough to be scrobbled it will return in newPlays
const newPlays = this.processRecentPlays(plays);
// finally, we return new plays and MS checks to see if they have been previously seen
// before signalling to Clients that they can be scrobbled
return newPlays;
}
}
```
Congratulations! Your `CoolPlayerSource` has been minimally implemented and can now be used in multi-scrobbler.
## Further Implementation
### Backlog
To have your Source try to scrobble "missed" tracks when MS starts up the Source's service must be able to provide:
* track information
* timestamp of when the track was played
In your Source implement [`getBackloggedPlays`](https://github.com/FoxxMD/multi-scrobbler/blob/master/src/backend/sources/AbstractSource.ts#L235) and set setting in constructor indicating it has backlogging capabilities:
throw new Error('Error occurred while getting recently played', {cause: e});
}
}
}
```
### Other Source Types
There are some scenarios where polling and/or state machine is not the right tool to handle determining if incoming data should be scrobbled:
* The Source service handles scrobble threshold internally, the data being received should always be scrobbled (WebScrobbler, Plex, Tautulli, Listenbrainz, Last.fm)
* You prefer to handle the scrobble determination yourself
#### Music History Source
If the Source is still polling but the track returned should always be scrobbled if not already seen IE the Source service is a **music history source** (Listenbrainz, Last.fm), rather than a music player, then simply indicate to MS the source of truth type by setting it in the constructor. The state machine will always return a track if it is new and not seen, regardless of how recently it was seen:
import { SOURCE_SOT } from "../../core/Atomic.js";
// ...
export default class CoolPlayerSource extends MemorySource {
constructor(/* ... */) {
super(/* ... */);
// ...
// tell MS it should immediately scrobble any new, unseen tracks from the upstream service
this.playerSourceOfTruth = SOURCE_SOT.HISTORY;
}
}
```
#### Non-Polling Source
**Ingress** Sources (like Plex, Tautulli, Webscrobbler, Jellyfin) do not having a polling mechanism because the upstream service contacts MS when there is an event, rather than MS contacting the upstream service.
For these Sources you will need to implement endpoints in `src/service/api.ts` and corresponding files. See the existing Sources in the project as references for how to do this.
You may still wish to use the state machine `MemorySource` (like Jellyfin) if the events received are not "scrobble" events but instead of implementing `getRecentlyPlayed` you will implement your own function in your Source class, like `handle()`, that receives data and then uses `processRecentPlays`.
After new plays have been determined see the next section for how to scrobble...
#### Basic Source
At the core of a Source that implements `AbstractSource`'s functionality is the ability to **Discover** and **Scrobble** plays.
These functions are not seen in the MVP `CoolPlayerSource` because they are automatically done by the polling functionality after being returned from `getRecentlyPlayed`.
##### Discovery
A Source keeps track of all the "plays" that are determined to be valid for scrobbling. When a play is valid it is checked to see if it has already been "discovered" by comparing the track info and timestamp of the play against already discovered plays. This prevents duplicate scrobbling by using the Source's own data and simplifies scrobbling for Sources by allowing your implementation to "always" ingest track data without having to worry about whether its new or not -- `AbstractSource` and `discover()` will take care of that for you.
This additionally will be surfaced to the user in the Dashboard in the "Tracks Discovered" page.
##### Scrobbling
After a play is verified to be discovered we can then scrobble it. This will emit the plays to the ScrobbleClients service which then disseminates the play to all Clients that were configured to listen in the Source's config.
If your service only emits an event when a play is scrobbled you can _technically_ skip using `discover()` but it is good practice to use it unless you have a very good reason not to.
:::note
Using `scrobble()` does not guarantee a track is actually scrobbled! The Scrobble Clients also check the play against their own "recently scrobbled" list to prevent duplicates.
These steps are for building the flatpak from source. If you want to install the application normally then [get it through flathub](../installation/installation.mdx#flatpak)
:::
The final build repo for the flathub version can be found at [flathub/io.github.foxxmd.multiscrobbler](https://github.com/flathub/io.github.foxxmd.multiscrobbler)
Set the `branch``tag` or `commit` to use for MS in the `source` section of [`io.github.foxxmd.multiscrobbler.yml`](https://github.com/FoxxMD/multi-scrobbler/blob/master/flatpak/io.github.foxxmd.multiscrobbler.yml)
## Generate Sources and Build
### Use Setup Script
A convenience bash script is provided that automates generating offline sources and building the flatpak app for you. This is the recommend method to use.
Located in the project at [`flatpak/setup.sh`](https://github.com/FoxxMD/multi-scrobbler/blob/master/flatpak/setup.sh), run it from the `flatpak` directory with this syntax:
```shell
./setup.sh -o -b /path/to/flatpak/build/dir
```
```
Args:
-o => Delete and overwrite any existing generated sources
-b => The absolute path to the directory that should be used for flatpak build/artifacts. If not defined will use `CWD/build`
-y => Proceed without confirming settings
```
### Manual Setup
If you cannot use `setup.sh` follow the below to manually generate sources and build the flatpak app:
<details>
<summary>Instructions</summary>
#### Use `flatpak-node-generator` to generate sources
First, [make sure `node_modules` is deleted or empty](https://github.com/flatpak/flatpak-builder-tools/issues/354#issuecomment-1478518442) for both the project and `docsite` directories.
Then, from MS project root:
```shell title="PROJECT_DIR"
flatpak-node-generator npm package-lock.json
```
Move `generated-sources.json` into `PROJECT_DIR/flatpak` and rename `generated-sources.0.json`
Append `--install --user` to the above command to have the app installed immediately.
::::
</details>
# Run App
If built with `--install --user` (default when using `setup.sh`) you can now run MS using the command
```shell
flatpak run -u io.github.foxxmd.multiscrobbler
```
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
FoxxMD
