Magento 2 Rest Api is clearing cache too frequently on product updates

Question

We have an issue with Magento 2's Api regarding the full page cache and updates from the rest API from our ERP system. The ERP is constantly pushing inventory, stock and product updates through the API and in turn this flushes the cache for every product update resulting in a consistently non-cached website. We tried to wrap around the FlushCacheByTags class to prevent rest calls from flushing the cache. This seems to be getting hit but the cache is still being cleared. Here is the class override:

<?php
/**
 *
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
namespace Namespace\Module\Magento\Framework\App\Cache;

/**
* Automatic cache cleaner plugin
*/
class FlushCacheByTags extends 
\Magento\Framework\App\Cache\FlushCacheByTags
{

/**
 * @var \Psr\Log\LoggerInterface
 */
protected $_logger;

/**
 * FlushCacheByTags constructor.
 * @param \Magento\Framework\App\Cache\Type\FrontendPool $cachePool
 * @param \Magento\Framework\App\Cache\StateInterface $cacheState
 * @param array $cacheList
 * @param null $tagResolver
 * @param \Psr\Log\LoggerInterface $_logger
 */
public function __construct(
    \Magento\Framework\App\Cache\Type\FrontendPool $cachePool,
    \Magento\Framework\App\Cache\StateInterface $cacheState,
    array $cacheList,
    \Psr\Log\LoggerInterface $_logger,
    $tagResolver = null

)
{
    parent::__construct($cachePool, $cacheState, $cacheList, $tagResolver);
    $this->_logger = $_logger;
}

/**
 * Clean cache on save object
 *
 * @param \Magento\Framework\Model\ResourceModel\AbstractResource $subject
 * @param \Closure $proceed
 * @param \Magento\Framework\Model\AbstractModel $object
 * @return \Magento\Framework\Model\ResourceModel\AbstractResource
 * @SuppressWarnings(PHPMD.UnusedFormalParameter)
 */
public function aroundSave(
    \Magento\Framework\Model\ResourceModel\AbstractResource $subject,
    \Closure $proceed,
    \Magento\Framework\Model\AbstractModel $object
) {
    $this->_logger->debug('CACHE SAVE - instance of: '. print_r(get_class($object),true));

    if (
            // is instance of
            /* @var $object \Magento\Catalog\Model\Product */
            $object instanceof \Magento\Catalog\Model\Product
        && (
            // is rest api request
            isset($_SERVER['REQUEST_URI']) && strpos($_SERVER['REQUEST_URI'],'/rest') === 0
            )
        && (
            // has data
            !empty($object->getData())
        )
    ) {
        $this->_logger->debug('Cache NOT flushed from API SKU#: '.print_r($object->getSku(),true));
        return $proceed($object);
    }

    return parent::aroundSave($subject, $proceed, $object);
}

/**
 * Clean cache on delete object
 *
 * @param \Magento\Framework\Model\ResourceModel\AbstractResource $subject
 * @param \Closure $proceed
 * @param \Magento\Framework\Model\AbstractModel $object
 * @return \Magento\Framework\Model\ResourceModel\AbstractResource
 * @SuppressWarnings(PHPMD.UnusedFormalParameter)
 */
public function aroundDelete(
    \Magento\Framework\Model\ResourceModel\AbstractResource $subject,
    \Closure $proceed,
    \Magento\Framework\Model\AbstractModel $object
) {
    $this->_logger->debug('CACHE DELETE - instance of: '. print_r(get_class($object),true));

    if (
        // is instance of
        /* @var $object \Magento\Catalog\Model\Product */
        $object instanceof \Magento\Catalog\Model\Product
        && (
            // is rest api request

            isset($_SERVER['REQUEST_URI']) && strpos($_SERVER['REQUEST_URI'],'/rest') === 0
        )
    ){
        $this->_logger->debug('Cache NOT flushed from API SKU#: '.print_r($object->getSku(),true));
        return $proceed($object);
    }
    return parent::aroundDelete($subject, $proceed, $object);
}
}

Are we missing something here? We would like to be able to push data to the database via the API and handle the flush on our own accord not every tima a product is updated via the API. We have implement a cache warmer to try and counter act the flushes but it can not keep up with the frequency at which the cache is flushed.

Here is what I am seeing in the logs on a product update from the API

[2017-06-09 21:26:05] report.DEBUG: CACHE SAVE - instance of: 
Magento\Catalog\Model\Product\Interceptor {"is_exception":false} []

[2017-06-09 21:26:05] report.DEBUG: Cache NOT flushed from API SKU#: 
270876 {"is_exception":false} []

[2017-06-09 21:26:05] report.DEBUG: cache_invalidate {"method":"PUT","url":"https://obscuredforsecurity.com/rest/all/V1/products/270876","invalidateInfo":{"tags":["catalog_product_0"],"mode":"matchingAnyTag"},"is_exception":false} []

Answer 1

to fix this we created an override for \Magento\PageCache\Observer\FlushCacheByTags

Here is the basics of the execute method. We ran into an issue where there was scenarios in which we actually did want the API to flush the cache so we implemented a "FlushNow" header and if that header is set in the request than we allow the cache to be flushed as normal. Also we had to implement similar features to prevent the saving of the categories and products in the admin from auto flushing the cache and acl to block out marketing department from access to the full page cache :).

** We use Mirasvits cache warming extension to warm our cache and they also implement similar features in their extension to prevent cache from being flushed from the admin (Not API). https://mirasvit.com/magento-2-extensions/full-page-cache-warmer.html

    /**
 * overide for \Magento\PageCache\Observer\FlushCacheByTags
 * Log calls to cache instead of clearing it
 *
 * @param Observer $observer
 * @return void
 */
public function execute(Observer $observer)
{
    if ($this->_config->getType() == Config::BUILT_IN && $this->_config->isEnabled()) {
        /** @noinspection PhpUndefinedMethodInspection */
        $object = $observer->getEvent()->getObject();
        if (!is_object($object)) {
            return;
        }
        $tags = $this->_tagResolver->getTags($object);
        $flush = $this->request->getHeaders('flushNow');
        $flush = $flush ? $flush->getFieldValue() : false;

        if (!empty($tags)) {
            if (
                $flush == false && (
                    (bool) $this->_scopeConfig->getValue(self::BLOCK_ALL)
                    || (
                        (bool) $this->_scopeConfig->getValue(self::BLOCK_API_CRON)
                        && (
                            !isset($_SERVER['REQUEST_URI'])
                            || preg_match('/^\/rest\/all\/V1/', $_SERVER['REQUEST_URI'])
                        )
                    )
                )
            ) {
                // Blocked entirely, Magento cron, or API request - Don't flush
                if ((bool) $this->_scopeConfig->getValue(self::DEBUG_LOG)) {
                    $this->writeToLog($tags);
                }
            }  else {
                if($flush){
                    $this->eventLogger->saveLog('cache_management','clean',json_encode($tags),true, 'api_cache_flush', true);
                }
                $this->_fullPageCache->clean(\Zend_Cache::CLEANING_MODE_MATCHING_ANY_TAG, array_unique($tags));
            }
        }
    }
}