Skip to content

fix(tweets): limit publishing of tweets based on date #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tigranstream wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(tweets): limit publishing of tweets based on date #88

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a9b6824
fix(tweets): limit publishing of tweets based on date
tigranavetis May 19, 2021
ec506e2
fix(tweets): add missing semicolon
tigranstream May 24, 2021
d179b9b
fix(tweets): remove unnecessary constructor
tigranstream May 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/configs/twitter.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,8 @@ module.exports = {
retweets: false,
skipValidAccounts: false,
},
feed_filters: {
date: 3600,
},
},
};
19 changes: 18 additions & 1 deletion src/services/twitter.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ const { HttpStatusError } = require('common-errors');
const {
isObject, isString, conforms, merge, find, isNil,
} = require('lodash');
const moment = require('moment');

const Notifier = require('./notifier');
const { transform, TYPE_TWEET } = require('../utils/response');
Expand Down Expand Up @@ -45,6 +46,13 @@ function streamFilterOptions(config) {
return merge({}, STREAM_FILTERS_DEFAULTS, config.stream_filters);
}

function feedFilters(config) {
const FEED_FILTERS_DEFAULTS = {
date: 3600,
};
return merge({}, FEED_FILTERS_DEFAULTS, config.feed_filters);
}

/**
* @property {TwitterClient} client
* @property {array} listeners
Expand Down Expand Up @@ -195,6 +203,7 @@ class Twitter {
this.client = new TwitterClient(config);
this.listener = null;
this.filterOptions = streamFilterOptions(config);
this.feedFilterOptions = feedFilters(config);
this.storage = storage;
this.logger = logger.child({ namespace: '@social/twitter' });
this._destroyed = false;
Expand Down Expand Up @@ -405,7 +414,15 @@ class Twitter {
const saved = await this.storage
.twitterStatuses()
.save(tweet);
this.publish(saved);

// Moment throws: value provided is not in a recognized RFC2822 or ISO format, that's why Date constructor is used.
const tweetDate = moment(new Date(tweet.date));
Copy link
Contributor

@BrRenat BrRenat May 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you show an example of tweet.date?

Copy link

@tigranavetis tigranavetis May 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BrRenat I took an example value from here:
https://developer.twitter.com/en/docs/twitter-api/v1/data-dictionary/object-model/tweet
"created_at": "Wed Oct 10 20:19:24 +0000 2018"

Copy link

@tigranavetis tigranavetis May 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed, however I got some warnings for the specified date:

Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged and will be removed in an upcoming major release. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.

Copy link
Contributor

@BrRenat BrRenat May 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JFYI
If you know the format of an input string, you can use that to parse a moment.
moment('24/12/2019 09:15:00', "DD MM YYYY hh:mm:ss");

const now = moment();
const diff = now.diff(tweetDate, 'seconds');

if (diff <= this.feedFilterOptions.date) {
this.publish(saved);
}
return saved;
} catch (err) {
this.logger.warn({ err }, 'failed to save tweet');
Expand Down